WIP image proxy

Fixes #231

.gitignore

@@ -10,6 +10,7 @@ app/public/uploads
 app/public/thumbnails
 celerybeat-schedule
 /data
+.idea
 
 # Created by https://www.gitignore.io/api/linux,macos,python,windows
 

Dockerfile

@@ -8,8 +8,8 @@ WORKDIR /home/cdb
 RUN mkdir /var/cdb
 RUN chown -R cdb:cdb /var/cdb
 
-COPY requirements.txt requirements.txt
-RUN pip install -r requirements.txt
+COPY requirements.lock.txt requirements.lock.txt
+RUN pip install -r requirements.lock.txt
 RUN pip install gunicorn
 
 COPY utils utils

LICENSE.md

@@ -0,0 +1,660 @@
+### GNU AFFERO GENERAL PUBLIC LICENSE
+
+Version 3, 19 November 2007
+
+Copyright (C) 2007 Free Software Foundation, Inc.
+<https://fsf.org/>
+
+Everyone is permitted to copy and distribute verbatim copies of this
+license document, but changing it is not allowed.
+
+### Preamble
+
+The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.
+
+The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works. By contrast,
+our General Public Licenses are intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains
+free software for all its users.
+
+When we speak of free software, we are referring to freedom, not
+price. Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.
+
+A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate. Many developers of free software are heartened and
+encouraged by the resulting cooperation. However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.
+
+The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community. It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server. Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.
+
+An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals. This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing
+under this license.
+
+The precise terms and conditions for copying, distribution and
+modification follow.
+
+### TERMS AND CONDITIONS
+
+#### 0. Definitions.
+
+"This License" refers to version 3 of the GNU Affero General Public
+License.
+
+"Copyright" also means copyright-like laws that apply to other kinds
+of works, such as semiconductor masks.
+
+"The Program" refers to any copyrightable work licensed under this
+License. Each licensee is addressed as "you". "Licensees" and
+"recipients" may be individuals or organizations.
+
+To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of
+an exact copy. The resulting work is called a "modified version" of
+the earlier work or a work "based on" the earlier work.
+
+A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy. Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies. Mere interaction with a user
+through a computer network, with no transfer of a copy, is not
+conveying.
+
+An interactive user interface displays "Appropriate Legal Notices" to
+the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License. If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+#### 1. Source Code.
+
+The "source code" for a work means the preferred form of the work for
+making modifications to it. "Object code" means any non-source form of
+a work.
+
+A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form. A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities. However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work. For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+The Corresponding Source need not include anything that users can
+regenerate automatically from other parts of the Corresponding Source.
+
+The Corresponding Source for a work in source code form is that same
+work.
+
+#### 2. Basic Permissions.
+
+All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met. This License explicitly affirms your unlimited
+permission to run the unmodified Program. The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work. This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+You may make, run and propagate covered works that you do not convey,
+without conditions so long as your license otherwise remains in force.
+You may convey covered works to others for the sole purpose of having
+them make modifications exclusively for you, or provide you with
+facilities for running those works, provided that you comply with the
+terms of this License in conveying all material for which you do not
+control copyright. Those thus making or running the covered works for
+you must do so exclusively on your behalf, under your direction and
+control, on terms that prohibit them from making any copies of your
+copyrighted material outside their relationship with you.
+
+Conveying under any other circumstances is permitted solely under the
+conditions stated below. Sublicensing is not allowed; section 10 makes
+it unnecessary.
+
+#### 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such
+circumvention is effected by exercising rights under this License with
+respect to the covered work, and you disclaim any intention to limit
+operation or modification of the work as a means of enforcing, against
+the work's users, your or third parties' legal rights to forbid
+circumvention of technological measures.
+
+#### 4. Conveying Verbatim Copies.
+
+You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+#### 5. Conveying Modified Source Versions.
+
+You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these
+conditions:
+
+-   a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+-   b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under
+    section 7. This requirement modifies the requirement in section 4
+    to "keep intact all notices".
+-   c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy. This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged. This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+-   d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit. Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+#### 6. Conveying Non-Source Forms.
+
+You may convey a covered work in object code form under the terms of
+sections 4 and 5, provided that you also convey the machine-readable
+Corresponding Source under the terms of this License, in one of these
+ways:
+
+-   a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+-   b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the Corresponding
+    Source from a network server at no charge.
+-   c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source. This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+-   d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge. You need not require recipients to copy the
+    Corresponding Source along with the object code. If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source. Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+-   e) Convey the object code using peer-to-peer transmission,
+    provided you inform other peers where the object code and
+    Corresponding Source of the work are being offered to the general
+    public at no charge under subsection 6d.
+
+A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal,
+family, or household purposes, or (2) anything designed or sold for
+incorporation into a dwelling. In determining whether a product is a
+consumer product, doubtful cases shall be resolved in favor of
+coverage. For a particular product received by a particular user,
+"normally used" refers to a typical or common use of that class of
+product, regardless of the status of the particular user or of the way
+in which the particular user actually uses, or expects or is expected
+to use, the product. A product is a consumer product regardless of
+whether the product has substantial commercial, industrial or
+non-consumer uses, unless such uses represent the only significant
+mode of use of the product.
+
+"Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to
+install and execute modified versions of a covered work in that User
+Product from a modified version of its Corresponding Source. The
+information must suffice to ensure that the continued functioning of
+the modified object code is in no case prevented or interfered with
+solely because modification has been made.
+
+If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information. But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or
+updates for a work that has been modified or installed by the
+recipient, or for the User Product in which it has been modified or
+installed. Access to a network may be denied when the modification
+itself materially and adversely affects the operation of the network
+or violates the rules and protocols for communication across the
+network.
+
+Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+#### 7. Additional Terms.
+
+"Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law. If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it. (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.) You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders
+of that material) supplement the terms of this License with terms:
+
+-   a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+-   b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+-   c) Prohibiting misrepresentation of the origin of that material,
+    or requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+-   d) Limiting the use for publicity purposes of names of licensors
+    or authors of the material; or
+-   e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+-   f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions
+    of it) with contractual assumptions of liability to the recipient,
+    for any liability that these contractual assumptions directly
+    impose on those licensors and authors.
+
+All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10. If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term. If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions; the
+above requirements apply either way.
+
+#### 8. Termination.
+
+You may not propagate or modify a covered work except as expressly
+provided under this License. Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+However, if you cease all violation of this License, then your license
+from a particular copyright holder is reinstated (a) provisionally,
+unless and until the copyright holder explicitly and finally
+terminates your license, and (b) permanently, if the copyright holder
+fails to notify you of the violation by some reasonable means prior to
+60 days after the cessation.
+
+Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License. If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+#### 9. Acceptance Not Required for Having Copies.
+
+You are not required to accept this License in order to receive or run
+a copy of the Program. Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance. However,
+nothing other than this License grants you permission to propagate or
+modify any covered work. These actions infringe copyright if you do
+not accept this License. Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+#### 10. Automatic Licensing of Downstream Recipients.
+
+Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License. You are not responsible
+for enforcing compliance by third parties with this License.
+
+An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations. If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License. For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+#### 11. Patents.
+
+A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based. The
+work thus licensed is called the contributor's "contributor version".
+
+A contributor's "essential patent claims" are all patent claims owned
+or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version. For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement). To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients. "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+A patent license is "discriminatory" if it does not include within the
+scope of its coverage, prohibits the exercise of, or is conditioned on
+the non-exercise of one or more of the rights that are specifically
+granted under this License. You may not convey a covered work if you
+are a party to an arrangement with a third party that is in the
+business of distributing software, under which you make payment to the
+third party based on the extent of your activity of conveying the
+work, and under which the third party grants, to any of the parties
+who would receive the covered work from you, a discriminatory patent
+license (a) in connection with copies of the covered work conveyed by
+you (or copies made from those copies), or (b) primarily for and in
+connection with specific products or compilations that contain the
+covered work, unless you entered into that arrangement, or that patent
+license was granted, prior to 28 March 2007.
+
+Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+#### 12. No Surrender of Others' Freedom.
+
+If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License. If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under
+this License and any other pertinent obligations, then as a
+consequence you may not convey it at all. For example, if you agree to
+terms that obligate you to collect a royalty for further conveying
+from those to whom you convey the Program, the only way you could
+satisfy both those terms and this License would be to refrain entirely
+from conveying the Program.
+
+#### 13. Remote Network Interaction; Use with the GNU General Public License.
+
+Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your
+version supports such interaction) an opportunity to receive the
+Corresponding Source of your version by providing access to the
+Corresponding Source from a network server at no charge, through some
+standard or customary means of facilitating copying of software. This
+Corresponding Source shall include the Corresponding Source for any
+work covered by version 3 of the GNU General Public License that is
+incorporated pursuant to the following paragraph.
+
+Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU General Public License into a single
+combined work, and to convey the resulting work. The terms of this
+License will continue to apply to the part which is the covered work,
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.
+
+#### 14. Revised Versions of this License.
+
+The Free Software Foundation may publish revised and/or new versions
+of the GNU Affero General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+Each version is given a distinguishing version number. If the Program
+specifies that a certain numbered version of the GNU Affero General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation. If the Program does not specify a version number of the
+GNU Affero General Public License, you may choose any version ever
+published by the Free Software Foundation.
+
+If the Program specifies that a proxy can decide which future versions
+of the GNU Affero General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+Later license versions may give you additional or different
+permissions. However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+#### 15. Disclaimer of Warranty.
+
+THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT
+WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND
+PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE
+DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR
+CORRECTION.
+
+#### 16. Limitation of Liability.
+
+IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR
+CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES
+ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT
+NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR
+LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM
+TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER
+PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
+
+#### 17. Interpretation of Sections 15 and 16.
+
+If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+END OF TERMS AND CONDITIONS
+
+### How to Apply These Terms to Your New Programs
+
+If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these
+terms.
+
+To do so, attach the following notices to the program. It is safest to
+attach them to the start of each source file to most effectively state
+the exclusion of warranty; and each file should have at least the
+"copyright" line and a pointer to where the full notice is found.
+
+        <one line to give the program's name and a brief idea of what it does.>
+        Copyright (C) <year>  <name of author>
+
+        This program is free software: you can redistribute it and/or modify
+        it under the terms of the GNU Affero General Public License as
+        published by the Free Software Foundation, either version 3 of the
+        License, or (at your option) any later version.
+
+        This program is distributed in the hope that it will be useful,
+        but WITHOUT ANY WARRANTY; without even the implied warranty of
+        MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+        GNU Affero General Public License for more details.
+
+        You should have received a copy of the GNU Affero General Public License
+        along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper
+mail.
+
+If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source. For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code. There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for
+the specific requirements.
+
+You should also get your employer (if you work as a programmer) or
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary. For more information on this, and how to apply and follow
+the GNU AGPL, see <https://www.gnu.org/licenses/>.

LICENSE.txt

@@ -1,674 +0,0 @@
-GNU GENERAL PUBLIC LICENSE
-   Version 3, 29 June 2007
-
-Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
-Everyone is permitted to copy and distribute verbatim copies
-of this license document, but changing it is not allowed.
-
-		Preamble
-
-The GNU General Public License is a free, copyleft license for
-software and other kinds of works.
-
-The licenses for most software and other practical works are designed
-to take away your freedom to share and change the works.  By contrast,
-the GNU General Public License is intended to guarantee your freedom to
-share and change all versions of a program--to make sure it remains free
-software for all its users.  We, the Free Software Foundation, use the
-GNU General Public License for most of our software; it applies also to
-any other work released this way by its authors.  You can apply it to
-your programs, too.
-
-When we speak of free software, we are referring to freedom, not
-price.  Our General Public Licenses are designed to make sure that you
-have the freedom to distribute copies of free software (and charge for
-them if you wish), that you receive source code or can get it if you
-want it, that you can change the software or use pieces of it in new
-free programs, and that you know you can do these things.
-
-To protect your rights, we need to prevent others from denying you
-these rights or asking you to surrender the rights.  Therefore, you have
-certain responsibilities if you distribute copies of the software, or if
-you modify it: responsibilities to respect the freedom of others.
-
-For example, if you distribute copies of such a program, whether
-gratis or for a fee, you must pass on to the recipients the same
-freedoms that you received.  You must make sure that they, too, receive
-or can get the source code.  And you must show them these terms so they
-know their rights.
-
-Developers that use the GNU GPL protect your rights with two steps:
-(1) assert copyright on the software, and (2) offer you this License
-giving you legal permission to copy, distribute and/or modify it.
-
-For the developers' and authors' protection, the GPL clearly explains
-that there is no warranty for this free software.  For both users' and
-authors' sake, the GPL requires that modified versions be marked as
-changed, so that their problems will not be attributed erroneously to
-authors of previous versions.
-
-Some devices are designed to deny users access to install or run
-modified versions of the software inside them, although the manufacturer
-can do so.  This is fundamentally incompatible with the aim of
-protecting users' freedom to change the software.  The systematic
-pattern of such abuse occurs in the area of products for individuals to
-use, which is precisely where it is most unacceptable.  Therefore, we
-have designed this version of the GPL to prohibit the practice for those
-products.  If such problems arise substantially in other domains, we
-stand ready to extend this provision to those domains in future versions
-of the GPL, as needed to protect the freedom of users.
-
-Finally, every program is threatened constantly by software patents.
-States should not allow patents to restrict development and use of
-software on general-purpose computers, but in those that do, we wish to
-avoid the special danger that patents applied to a free program could
-make it effectively proprietary.  To prevent this, the GPL assures that
-patents cannot be used to render the program non-free.
-
-The precise terms and conditions for copying, distribution and
-modification follow.
-
-   TERMS AND CONDITIONS
-
-0. Definitions.
-
-"This License" refers to version 3 of the GNU General Public License.
-
-"Copyright" also means copyright-like laws that apply to other kinds of
-works, such as semiconductor masks.
-
-"The Program" refers to any copyrightable work licensed under this
-License.  Each licensee is addressed as "you".  "Licensees" and
-"recipients" may be individuals or organizations.
-
-To "modify" a work means to copy from or adapt all or part of the work
-in a fashion requiring copyright permission, other than the making of an
-exact copy.  The resulting work is called a "modified version" of the
-earlier work or a work "based on" the earlier work.
-
-A "covered work" means either the unmodified Program or a work based
-on the Program.
-
-To "propagate" a work means to do anything with it that, without
-permission, would make you directly or secondarily liable for
-infringement under applicable copyright law, except executing it on a
-computer or modifying a private copy.  Propagation includes copying,
-distribution (with or without modification), making available to the
-public, and in some countries other activities as well.
-
-To "convey" a work means any kind of propagation that enables other
-parties to make or receive copies.  Mere interaction with a user through
-a computer network, with no transfer of a copy, is not conveying.
-
-An interactive user interface displays "Appropriate Legal Notices"
-to the extent that it includes a convenient and prominently visible
-feature that (1) displays an appropriate copyright notice, and (2)
-tells the user that there is no warranty for the work (except to the
-extent that warranties are provided), that licensees may convey the
-work under this License, and how to view a copy of this License.  If
-the interface presents a list of user commands or options, such as a
-menu, a prominent item in the list meets this criterion.
-
-1. Source Code.
-
-The "source code" for a work means the preferred form of the work
-for making modifications to it.  "Object code" means any non-source
-form of a work.
-
-A "Standard Interface" means an interface that either is an official
-standard defined by a recognized standards body, or, in the case of
-interfaces specified for a particular programming language, one that
-is widely used among developers working in that language.
-
-The "System Libraries" of an executable work include anything, other
-than the work as a whole, that (a) is included in the normal form of
-packaging a Major Component, but which is not part of that Major
-Component, and (b) serves only to enable use of the work with that
-Major Component, or to implement a Standard Interface for which an
-implementation is available to the public in source code form.  A
-"Major Component", in this context, means a major essential component
-(kernel, window system, and so on) of the specific operating system
-(if any) on which the executable work runs, or a compiler used to
-produce the work, or an object code interpreter used to run it.
-
-The "Corresponding Source" for a work in object code form means all
-the source code needed to generate, install, and (for an executable
-work) run the object code and to modify the work, including scripts to
-control those activities.  However, it does not include the work's
-System Libraries, or general-purpose tools or generally available free
-programs which are used unmodified in performing those activities but
-which are not part of the work.  For example, Corresponding Source
-includes interface definition files associated with source files for
-the work, and the source code for shared libraries and dynamically
-linked subprograms that the work is specifically designed to require,
-such as by intimate data communication or control flow between those
-subprograms and other parts of the work.
-
-The Corresponding Source need not include anything that users
-can regenerate automatically from other parts of the Corresponding
-Source.
-
-The Corresponding Source for a work in source code form is that
-same work.
-
-2. Basic Permissions.
-
-All rights granted under this License are granted for the term of
-copyright on the Program, and are irrevocable provided the stated
-conditions are met.  This License explicitly affirms your unlimited
-permission to run the unmodified Program.  The output from running a
-covered work is covered by this License only if the output, given its
-content, constitutes a covered work.  This License acknowledges your
-rights of fair use or other equivalent, as provided by copyright law.
-
-You may make, run and propagate covered works that you do not
-convey, without conditions so long as your license otherwise remains
-in force.  You may convey covered works to others for the sole purpose
-of having them make modifications exclusively for you, or provide you
-with facilities for running those works, provided that you comply with
-the terms of this License in conveying all material for which you do
-not control copyright.  Those thus making or running the covered works
-for you must do so exclusively on your behalf, under your direction
-and control, on terms that prohibit them from making any copies of
-your copyrighted material outside their relationship with you.
-
-Conveying under any other circumstances is permitted solely under
-the conditions stated below.  Sublicensing is not allowed; section 10
-makes it unnecessary.
-
-3. Protecting Users' Legal Rights From Anti-Circumvention Law.
-
-No covered work shall be deemed part of an effective technological
-measure under any applicable law fulfilling obligations under article
-11 of the WIPO copyright treaty adopted on 20 December 1996, or
-similar laws prohibiting or restricting circumvention of such
-measures.
-
-When you convey a covered work, you waive any legal power to forbid
-circumvention of technological measures to the extent such circumvention
-is effected by exercising rights under this License with respect to
-the covered work, and you disclaim any intention to limit operation or
-modification of the work as a means of enforcing, against the work's
-users, your or third parties' legal rights to forbid circumvention of
-technological measures.
-
-4. Conveying Verbatim Copies.
-
-You may convey verbatim copies of the Program's source code as you
-receive it, in any medium, provided that you conspicuously and
-appropriately publish on each copy an appropriate copyright notice;
-keep intact all notices stating that this License and any
-non-permissive terms added in accord with section 7 apply to the code;
-keep intact all notices of the absence of any warranty; and give all
-recipients a copy of this License along with the Program.
-
-You may charge any price or no price for each copy that you convey,
-and you may offer support or warranty protection for a fee.
-
-5. Conveying Modified Source Versions.
-
-You may convey a work based on the Program, or the modifications to
-produce it from the Program, in the form of source code under the
-terms of section 4, provided that you also meet all of these conditions:
-
-a) The work must carry prominent notices stating that you modified
-it, and giving a relevant date.
-
-b) The work must carry prominent notices stating that it is
-released under this License and any conditions added under section
-7.  This requirement modifies the requirement in section 4 to
-"keep intact all notices".
-
-c) You must license the entire work, as a whole, under this
-License to anyone who comes into possession of a copy.  This
-License will therefore apply, along with any applicable section 7
-additional terms, to the whole of the work, and all its parts,
-regardless of how they are packaged.  This License gives no
-permission to license the work in any other way, but it does not
-invalidate such permission if you have separately received it.
-
-d) If the work has interactive user interfaces, each must display
-Appropriate Legal Notices; however, if the Program has interactive
-interfaces that do not display Appropriate Legal Notices, your
-work need not make them do so.
-
-A compilation of a covered work with other separate and independent
-works, which are not by their nature extensions of the covered work,
-and which are not combined with it such as to form a larger program,
-in or on a volume of a storage or distribution medium, is called an
-"aggregate" if the compilation and its resulting copyright are not
-used to limit the access or legal rights of the compilation's users
-beyond what the individual works permit.  Inclusion of a covered work
-in an aggregate does not cause this License to apply to the other
-parts of the aggregate.
-
-6. Conveying Non-Source Forms.
-
-You may convey a covered work in object code form under the terms
-of sections 4 and 5, provided that you also convey the
-machine-readable Corresponding Source under the terms of this License,
-in one of these ways:
-
-a) Convey the object code in, or embodied in, a physical product
-(including a physical distribution medium), accompanied by the
-Corresponding Source fixed on a durable physical medium
-customarily used for software interchange.
-
-b) Convey the object code in, or embodied in, a physical product
-(including a physical distribution medium), accompanied by a
-written offer, valid for at least three years and valid for as
-long as you offer spare parts or customer support for that product
-model, to give anyone who possesses the object code either (1) a
-copy of the Corresponding Source for all the software in the
-product that is covered by this License, on a durable physical
-medium customarily used for software interchange, for a price no
-more than your reasonable cost of physically performing this
-conveying of source, or (2) access to copy the
-Corresponding Source from a network server at no charge.
-
-c) Convey individual copies of the object code with a copy of the
-written offer to provide the Corresponding Source.  This
-alternative is allowed only occasionally and noncommercially, and
-only if you received the object code with such an offer, in accord
-with subsection 6b.
-
-d) Convey the object code by offering access from a designated
-place (gratis or for a charge), and offer equivalent access to the
-Corresponding Source in the same way through the same place at no
-further charge.  You need not require recipients to copy the
-Corresponding Source along with the object code.  If the place to
-copy the object code is a network server, the Corresponding Source
-may be on a different server (operated by you or a third party)
-that supports equivalent copying facilities, provided you maintain
-clear directions next to the object code saying where to find the
-Corresponding Source.  Regardless of what server hosts the
-Corresponding Source, you remain obligated to ensure that it is
-available for as long as needed to satisfy these requirements.
-
-e) Convey the object code using peer-to-peer transmission, provided
-you inform other peers where the object code and Corresponding
-Source of the work are being offered to the general public at no
-charge under subsection 6d.
-
-A separable portion of the object code, whose source code is excluded
-from the Corresponding Source as a System Library, need not be
-included in conveying the object code work.
-
-A "User Product" is either (1) a "consumer product", which means any
-tangible personal property which is normally used for personal, family,
-or household purposes, or (2) anything designed or sold for incorporation
-into a dwelling.  In determining whether a product is a consumer product,
-doubtful cases shall be resolved in favor of coverage.  For a particular
-product received by a particular user, "normally used" refers to a
-typical or common use of that class of product, regardless of the status
-of the particular user or of the way in which the particular user
-actually uses, or expects or is expected to use, the product.  A product
-is a consumer product regardless of whether the product has substantial
-commercial, industrial or non-consumer uses, unless such uses represent
-the only significant mode of use of the product.
-
-"Installation Information" for a User Product means any methods,
-procedures, authorization keys, or other information required to install
-and execute modified versions of a covered work in that User Product from
-a modified version of its Corresponding Source.  The information must
-suffice to ensure that the continued functioning of the modified object
-code is in no case prevented or interfered with solely because
-modification has been made.
-
-If you convey an object code work under this section in, or with, or
-specifically for use in, a User Product, and the conveying occurs as
-part of a transaction in which the right of possession and use of the
-User Product is transferred to the recipient in perpetuity or for a
-fixed term (regardless of how the transaction is characterized), the
-Corresponding Source conveyed under this section must be accompanied
-by the Installation Information.  But this requirement does not apply
-if neither you nor any third party retains the ability to install
-modified object code on the User Product (for example, the work has
-been installed in ROM).
-
-The requirement to provide Installation Information does not include a
-requirement to continue to provide support service, warranty, or updates
-for a work that has been modified or installed by the recipient, or for
-the User Product in which it has been modified or installed.  Access to a
-network may be denied when the modification itself materially and
-adversely affects the operation of the network or violates the rules and
-protocols for communication across the network.
-
-Corresponding Source conveyed, and Installation Information provided,
-in accord with this section must be in a format that is publicly
-documented (and with an implementation available to the public in
-source code form), and must require no special password or key for
-unpacking, reading or copying.
-
-7. Additional Terms.
-
-"Additional permissions" are terms that supplement the terms of this
-License by making exceptions from one or more of its conditions.
-Additional permissions that are applicable to the entire Program shall
-be treated as though they were included in this License, to the extent
-that they are valid under applicable law.  If additional permissions
-apply only to part of the Program, that part may be used separately
-under those permissions, but the entire Program remains governed by
-this License without regard to the additional permissions.
-
-When you convey a copy of a covered work, you may at your option
-remove any additional permissions from that copy, or from any part of
-it.  (Additional permissions may be written to require their own
-removal in certain cases when you modify the work.)  You may place
-additional permissions on material, added by you to a covered work,
-for which you have or can give appropriate copyright permission.
-
-Notwithstanding any other provision of this License, for material you
-add to a covered work, you may (if authorized by the copyright holders of
-that material) supplement the terms of this License with terms:
-
-a) Disclaiming warranty or limiting liability differently from the
-terms of sections 15 and 16 of this License; or
-
-b) Requiring preservation of specified reasonable legal notices or
-author attributions in that material or in the Appropriate Legal
-Notices displayed by works containing it; or
-
-c) Prohibiting misrepresentation of the origin of that material, or
-requiring that modified versions of such material be marked in
-reasonable ways as different from the original version; or
-
-d) Limiting the use for publicity purposes of names of licensors or
-authors of the material; or
-
-e) Declining to grant rights under trademark law for use of some
-trade names, trademarks, or service marks; or
-
-f) Requiring indemnification of licensors and authors of that
-material by anyone who conveys the material (or modified versions of
-it) with contractual assumptions of liability to the recipient, for
-any liability that these contractual assumptions directly impose on
-those licensors and authors.
-
-All other non-permissive additional terms are considered "further
-restrictions" within the meaning of section 10.  If the Program as you
-received it, or any part of it, contains a notice stating that it is
-governed by this License along with a term that is a further
-restriction, you may remove that term.  If a license document contains
-a further restriction but permits relicensing or conveying under this
-License, you may add to a covered work material governed by the terms
-of that license document, provided that the further restriction does
-not survive such relicensing or conveying.
-
-If you add terms to a covered work in accord with this section, you
-must place, in the relevant source files, a statement of the
-additional terms that apply to those files, or a notice indicating
-where to find the applicable terms.
-
-Additional terms, permissive or non-permissive, may be stated in the
-form of a separately written license, or stated as exceptions;
-the above requirements apply either way.
-
-8. Termination.
-
-You may not propagate or modify a covered work except as expressly
-provided under this License.  Any attempt otherwise to propagate or
-modify it is void, and will automatically terminate your rights under
-this License (including any patent licenses granted under the third
-paragraph of section 11).
-
-However, if you cease all violation of this License, then your
-license from a particular copyright holder is reinstated (a)
-provisionally, unless and until the copyright holder explicitly and
-finally terminates your license, and (b) permanently, if the copyright
-holder fails to notify you of the violation by some reasonable means
-prior to 60 days after the cessation.
-
-Moreover, your license from a particular copyright holder is
-reinstated permanently if the copyright holder notifies you of the
-violation by some reasonable means, this is the first time you have
-received notice of violation of this License (for any work) from that
-copyright holder, and you cure the violation prior to 30 days after
-your receipt of the notice.
-
-Termination of your rights under this section does not terminate the
-licenses of parties who have received copies or rights from you under
-this License.  If your rights have been terminated and not permanently
-reinstated, you do not qualify to receive new licenses for the same
-material under section 10.
-
-9. Acceptance Not Required for Having Copies.
-
-You are not required to accept this License in order to receive or
-run a copy of the Program.  Ancillary propagation of a covered work
-occurring solely as a consequence of using peer-to-peer transmission
-to receive a copy likewise does not require acceptance.  However,
-nothing other than this License grants you permission to propagate or
-modify any covered work.  These actions infringe copyright if you do
-not accept this License.  Therefore, by modifying or propagating a
-covered work, you indicate your acceptance of this License to do so.
-
-10. Automatic Licensing of Downstream Recipients.
-
-Each time you convey a covered work, the recipient automatically
-receives a license from the original licensors, to run, modify and
-propagate that work, subject to this License.  You are not responsible
-for enforcing compliance by third parties with this License.
-
-An "entity transaction" is a transaction transferring control of an
-organization, or substantially all assets of one, or subdividing an
-organization, or merging organizations.  If propagation of a covered
-work results from an entity transaction, each party to that
-transaction who receives a copy of the work also receives whatever
-licenses to the work the party's predecessor in interest had or could
-give under the previous paragraph, plus a right to possession of the
-Corresponding Source of the work from the predecessor in interest, if
-the predecessor has it or can get it with reasonable efforts.
-
-You may not impose any further restrictions on the exercise of the
-rights granted or affirmed under this License.  For example, you may
-not impose a license fee, royalty, or other charge for exercise of
-rights granted under this License, and you may not initiate litigation
-(including a cross-claim or counterclaim in a lawsuit) alleging that
-any patent claim is infringed by making, using, selling, offering for
-sale, or importing the Program or any portion of it.
-
-11. Patents.
-
-A "contributor" is a copyright holder who authorizes use under this
-License of the Program or a work on which the Program is based.  The
-work thus licensed is called the contributor's "contributor version".
-
-A contributor's "essential patent claims" are all patent claims
-owned or controlled by the contributor, whether already acquired or
-hereafter acquired, that would be infringed by some manner, permitted
-by this License, of making, using, or selling its contributor version,
-but do not include claims that would be infringed only as a
-consequence of further modification of the contributor version.  For
-purposes of this definition, "control" includes the right to grant
-patent sublicenses in a manner consistent with the requirements of
-this License.
-
-Each contributor grants you a non-exclusive, worldwide, royalty-free
-patent license under the contributor's essential patent claims, to
-make, use, sell, offer for sale, import and otherwise run, modify and
-propagate the contents of its contributor version.
-
-In the following three paragraphs, a "patent license" is any express
-agreement or commitment, however denominated, not to enforce a patent
-(such as an express permission to practice a patent or covenant not to
-sue for patent infringement).  To "grant" such a patent license to a
-party means to make such an agreement or commitment not to enforce a
-patent against the party.
-
-If you convey a covered work, knowingly relying on a patent license,
-and the Corresponding Source of the work is not available for anyone
-to copy, free of charge and under the terms of this License, through a
-publicly available network server or other readily accessible means,
-then you must either (1) cause the Corresponding Source to be so
-available, or (2) arrange to deprive yourself of the benefit of the
-patent license for this particular work, or (3) arrange, in a manner
-consistent with the requirements of this License, to extend the patent
-license to downstream recipients.  "Knowingly relying" means you have
-actual knowledge that, but for the patent license, your conveying the
-covered work in a country, or your recipient's use of the covered work
-in a country, would infringe one or more identifiable patents in that
-country that you have reason to believe are valid.
-
-If, pursuant to or in connection with a single transaction or
-arrangement, you convey, or propagate by procuring conveyance of, a
-covered work, and grant a patent license to some of the parties
-receiving the covered work authorizing them to use, propagate, modify
-or convey a specific copy of the covered work, then the patent license
-you grant is automatically extended to all recipients of the covered
-work and works based on it.
-
-A patent license is "discriminatory" if it does not include within
-the scope of its coverage, prohibits the exercise of, or is
-conditioned on the non-exercise of one or more of the rights that are
-specifically granted under this License.  You may not convey a covered
-work if you are a party to an arrangement with a third party that is
-in the business of distributing software, under which you make payment
-to the third party based on the extent of your activity of conveying
-the work, and under which the third party grants, to any of the
-parties who would receive the covered work from you, a discriminatory
-patent license (a) in connection with copies of the covered work
-conveyed by you (or copies made from those copies), or (b) primarily
-for and in connection with specific products or compilations that
-contain the covered work, unless you entered into that arrangement,
-or that patent license was granted, prior to 28 March 2007.
-
-Nothing in this License shall be construed as excluding or limiting
-any implied license or other defenses to infringement that may
-otherwise be available to you under applicable patent law.
-
-12. No Surrender of Others' Freedom.
-
-If conditions are imposed on you (whether by court order, agreement or
-otherwise) that contradict the conditions of this License, they do not
-excuse you from the conditions of this License.  If you cannot convey a
-covered work so as to satisfy simultaneously your obligations under this
-License and any other pertinent obligations, then as a consequence you may
-not convey it at all.  For example, if you agree to terms that obligate you
-to collect a royalty for further conveying from those to whom you convey
-the Program, the only way you could satisfy both those terms and this
-License would be to refrain entirely from conveying the Program.
-
-13. Use with the GNU Affero General Public License.
-
-Notwithstanding any other provision of this License, you have
-permission to link or combine any covered work with a work licensed
-under version 3 of the GNU Affero General Public License into a single
-combined work, and to convey the resulting work.  The terms of this
-License will continue to apply to the part which is the covered work,
-but the special requirements of the GNU Affero General Public License,
-section 13, concerning interaction through a network will apply to the
-combination as such.
-
-14. Revised Versions of this License.
-
-The Free Software Foundation may publish revised and/or new versions of
-the GNU General Public License from time to time.  Such new versions will
-be similar in spirit to the present version, but may differ in detail to
-address new problems or concerns.
-
-Each version is given a distinguishing version number.  If the
-Program specifies that a certain numbered version of the GNU General
-Public License "or any later version" applies to it, you have the
-option of following the terms and conditions either of that numbered
-version or of any later version published by the Free Software
-Foundation.  If the Program does not specify a version number of the
-GNU General Public License, you may choose any version ever published
-by the Free Software Foundation.
-
-If the Program specifies that a proxy can decide which future
-versions of the GNU General Public License can be used, that proxy's
-public statement of acceptance of a version permanently authorizes you
-to choose that version for the Program.
-
-Later license versions may give you additional or different
-permissions.  However, no additional obligations are imposed on any
-author or copyright holder as a result of your choosing to follow a
-later version.
-
-15. Disclaimer of Warranty.
-
-THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
-APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
-HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
-OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
-THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
-IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
-ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
-
-16. Limitation of Liability.
-
-IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
-WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
-THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
-GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
-USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
-DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
-PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
-EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
-SUCH DAMAGES.
-
-17. Interpretation of Sections 15 and 16.
-
-If the disclaimer of warranty and limitation of liability provided
-above cannot be given local legal effect according to their terms,
-reviewing courts shall apply local law that most closely approximates
-an absolute waiver of all civil liability in connection with the
-Program, unless a warranty or assumption of liability accompanies a
-copy of the Program in return for a fee.
-
- END OF TERMS AND CONDITIONS
-
-How to Apply These Terms to Your New Programs
-
-If you develop a new program, and you want it to be of the greatest
-possible use to the public, the best way to achieve this is to make it
-free software which everyone can redistribute and change under these terms.
-
-To do so, attach the following notices to the program.  It is safest
-to attach them to the start of each source file to most effectively
-state the exclusion of warranty; and each file should have at least
-the "copyright" line and a pointer to where the full notice is found.
-
-<one line to give the program's name and a brief idea of what it does.>
-Copyright (C) <year>  <name of author>
-
-This program is free software: you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-Also add information on how to contact you by electronic and paper mail.
-
-If the program does terminal interaction, make it output a short
-notice like this when it starts in an interactive mode:
-
-<program>  Copyright (C) <year>  <name of author>
-This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
-This is free software, and you are welcome to redistribute it
-under certain conditions; type `show c' for details.
-
-The hypothetical commands `show w' and `show c' should show the appropriate
-parts of the General Public License.  Of course, your program's commands
-might be different; for a GUI interface, you would use an "about box".
-
-You should also get your employer (if you work as a programmer) or school,
-if any, to sign a "copyright disclaimer" for the program, if necessary.
-For more information on this, and how to apply and follow the GNU GPL, see
-<https://www.gnu.org/licenses/>.
-
-The GNU General Public License does not permit incorporating your program
-into proprietary programs.  If your program is a subroutine library, you
-may consider it more useful to permit linking proprietary applications with
-the library.  If this is what you want to do, use the GNU Lesser General
-Public License instead of this License.  But first, please read
-<https://www.gnu.org/licenses/why-not-lgpl.html>.

README.md

@@ -1,36 +1,65 @@
 # Content Database
+[![Build status](https://gitlab.com/minetest/contentdb/badges/master/pipeline.svg)](https://gitlab.com/minetest/contentdb/pipelines)
 
-Content database for Minetest mods, games, and more.
+Content database for Minetest mods, games, and more.\
+Developed by rubenwardy, license AGPLv3.0+.
 
-Developed by rubenwardy, license GPLv3.0+.
+See [Getting Started](docs/getting_started.md).
 
 ## How-tos
 
-Note: you should first read one of the guides on the [Github repo wiki](https://github.com/minetest/contentdb/wiki)
-
 ```sh
-# Run celery worker
-FLASK_CONFIG=../config.cfg celery -A app.tasks.celery worker
-
-# if sqlite
-python utils/setup.py -t
-rm db.sqlite && python setup.py -t && FLASK_CONFIG=../config.cfg FLASK_APP=app/__init__.py flask db stamp head
-
-# Create migration
-FLASK_CONFIG=../config.cfg FLASK_APP=app/__init__.py flask db migrate
-# Run migration
-FLASK_CONFIG=../config.cfg FLASK_APP=app/__init__.py flask db upgrade
-
-# Enter docker
-docker exec -it contentdb_app_1 bash
-
 # Hot/live reload (only works with FLASK_DEBUG=1)
 ./utils/reload.sh
 
-# Cold update a running version of CDB with minimal downtime
+# Cold update a running version of CDB with minimal downtime (production)
 ./utils/update.sh
+
+# Enter docker
+./utils/bash.sh
+
+# Run migrations
+./utils/run_migrations.sh
+
+# Create new migration
+./utils/create_migration.sh
 ```
 
+
+### VSCode: Setting up Linting
+
+* (optional) Install the [Docker extension](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker)
+* Install the [Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python)
+	* Click no to installing pylint (we don't want it to be installed outside of a virtual env)
+* Set up a virtual env
+	* Replace `psycopg2` with `psycopg2_binary` in requirements.txt (because postgresql won't be installed on the system)
+	* `python3 -m venv env`
+	* Click yes to prompt to select virtual env for workspace
+	* Click yes to any prompts about installing pylint
+	* `source env/bin/activate`
+	* `pip install -r requirements`
+	* `pip install pylint` (if a prompt didn't appear)
+	* Undo changes to requirements.txt
+
+### VSCode: Material Icon Folder Designations
+
+```json
+"material-icon-theme.folders.associations": {
+	"packages": "",
+	"tasks": "",
+	"api": "",
+	"meta": "",
+	"blueprints": "routes",
+	"scss": "sass",
+	"flatpages": "markdown",
+	"data": "temp",
+	"migrations": "archive",
+	"textures": "images",
+	"sounds": "audio"
+}
+```
+
+
 ## Database
 
 

app/__init__.py

@@ -1,22 +1,21 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21  rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
 from flask import *
-from flask_user import *
 from flask_gravatar import Gravatar
 import flask_menu as menu
 from flask_mail import Mail
@@ -24,11 +23,20 @@ from flask_github import GitHub
 from flask_wtf.csrf import CSRFProtect
 from flask_flatpages import FlatPages
 from flask_babel import Babel
+from flask_login import logout_user, current_user, LoginManager
 import os, redis
 
 app = Flask(__name__, static_folder="public/static")
 app.config["FLATPAGES_ROOT"] = "flatpages"
 app.config["FLATPAGES_EXTENSION"] = ".md"
+app.config["FLATPAGES_MARKDOWN_EXTENSIONS"] = ["fenced_code", "tables", "codehilite", 'toc']
+app.config["FLATPAGES_EXTENSION_CONFIG"] = {
+	"fenced_code": {},
+	"tables": {},
+	"codehilite": {
+		"linenums": "True"
+	}
+}
 app.config.from_pyfile(os.environ["FLASK_CONFIG"])
 
 r = redis.Redis.from_url(app.config["REDIS_URL"])
@@ -41,47 +49,53 @@ pages = FlatPages(app)
 babel = Babel(app)
 gravatar = Gravatar(app,
 		size=58,
-		rating='g',
-		default='mp',
+		rating="g",
+		default="mp",
 		force_default=False,
 		force_lower=False,
 		use_ssl=True,
 		base_url=None)
 
+login_manager = LoginManager()
+login_manager.init_app(app)
+login_manager.login_view = "users.login"
+
 from .sass import sass
 sass(app)
 
 
 if not app.debug and app.config["MAIL_UTILS_ERROR_SEND_TO"]:
-	from .maillogger import register_mail_error_handler
-	register_mail_error_handler(app, mail)
+	from .maillogger import build_handler
+	app.logger.addHandler(build_handler(app))
 
 
-from .markdown import init_app
+from app.utils.markdown import init_app
 init_app(app)
 
-
 # @babel.localeselector
 # def get_locale():
-# 	return request.accept_languages.best_match(app.config['LANGUAGES'].keys())
+# 	return request.accept_languages.best_match(app.config["LANGUAGES"].keys())
+
+from . import models, template_filters
+
+@login_manager.user_loader
+def load_user(user_id):
+	return models.User.query.filter_by(username=user_id).first()
 
-from . import models, tasks, template_filters
 
 from .blueprints import create_blueprints
 create_blueprints(app)
 
-from flask_login import logout_user
-
 @app.route("/uploads/<path:path>")
 def send_upload(path):
-	return send_from_directory(app.config['UPLOAD_DIR'], path)
+	return send_from_directory(app.config["UPLOAD_DIR"], path)
 
-@menu.register_menu(app, ".help", "Help", order=19, endpoint_arguments_constructor=lambda: { 'path': 'help' })
-@app.route('/<path:path>/')
+@menu.register_menu(app, ".help", "Help", order=19, endpoint_arguments_constructor=lambda: { "path": "help" })
+@app.route("/<path:path>/")
 def flatpage(path):
-    page = pages.get_or_404(path)
-    template = page.meta.get('template', 'flatpage.html')
-    return render_template(template, page=page)
+	page = pages.get_or_404(path)
+	template = page.meta.get("template", "flatpage.html")
+	return render_template(template, page=page)
 
 @app.before_request
 def check_for_ban():
@@ -89,7 +103,14 @@ def check_for_ban():
 		if current_user.rank == models.UserRank.BANNED:
 			flash("You have been banned.", "danger")
 			logout_user()
-			return redirect(url_for('user.login'))
+			return redirect(url_for("users.login"))
 		elif current_user.rank == models.UserRank.NOT_JOINED:
 			current_user.rank = models.UserRank.MEMBER
 			models.db.session.commit()
+
+from .utils import clearNotifications
+
+@app.before_request
+def check_for_notifications():
+	if current_user.is_authenticated:
+		clearNotifications(request.path)

app/blueprints/admin/__init__.py

@@ -1,17 +1,17 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
@@ -19,4 +19,4 @@ from flask import Blueprint
 
 bp = Blueprint("admin", __name__)
 
-from . import admin, licenseseditor, tagseditor, versioneditor
+from . import admin, audit, licenseseditor, tagseditor, versioneditor, warningseditor, email

app/blueprints/admin/admin.py

@@ -1,42 +1,47 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
+import os
+
+from celery import group
 from flask import *
-from flask_user import *
-import flask_menu as menu
-from . import bp
-from app.models import *
-from celery import uuid, group
-from app.tasks.importtasks import importRepoScreenshot, importAllDependencies, makeVCSRelease, checkZipRelease
-from app.tasks.forumtasks  import importTopicList, checkAllForumAccounts
+from flask_login import current_user, login_user
 from flask_wtf import FlaskForm
 from wtforms import *
-from app.utils import loginUser, rank_required, triggerNotif
-import datetime
+from wtforms.validators import InputRequired, Length
+
+from app.models import *
+from app.tasks.forumtasks import importTopicList, checkAllForumAccounts
+from app.tasks.importtasks import importRepoScreenshot, checkZipRelease, check_for_updates
+from app.utils import rank_required, addAuditLog, addNotification
+from . import bp
+
 
 @bp.route("/admin/", methods=["GET", "POST"])
 @rank_required(UserRank.ADMIN)
 def admin_page():
 	if request.method == "POST":
 		action = request.form["action"]
+
 		if action == "delstuckreleases":
 			PackageRelease.query.filter(PackageRelease.task_id != None).delete()
 			db.session.commit()
 			return redirect(url_for("admin.admin_page"))
+
 		elif action == "checkreleases":
 			releases = PackageRelease.query.filter(PackageRelease.url.like("/uploads/%")).all()
 
@@ -51,16 +56,35 @@ def admin_page():
 				import time
 				time.sleep(0.1)
 
-			return redirect(url_for("todo.view"))
+			return redirect(url_for("todo.view_editor"))
+
+		elif action == "reimportpackages":
+			tasks = []
+			for package in Package.query.filter(Package.state!=PackageState.DELETED).all():
+				release = package.releases.first()
+				if release:
+					zippath = release.url.replace("/uploads/", app.config["UPLOAD_DIR"])
+					tasks.append(checkZipRelease.s(release.id, zippath))
+
+			result = group(tasks).apply_async()
+
+			while not result.ready():
+				import time
+				time.sleep(0.1)
+
+			return redirect(url_for("todo.view_editor"))
+
 		elif action == "importmodlist":
 			task = importTopicList.delay()
 			return redirect(url_for("tasks.check", id=task.id, r=url_for("todo.topics")))
+
 		elif action == "checkusers":
 			task = checkAllForumAccounts.delay()
 			return redirect(url_for("tasks.check", id=task.id, r=url_for("admin.admin_page")))
+
 		elif action == "importscreenshots":
 			packages = Package.query \
-				.filter_by(soft_deleted=False) \
+				.filter(Package.state!=PackageState.DELETED) \
 				.outerjoin(PackageScreenshot, Package.id==PackageScreenshot.package_id) \
 				.filter(PackageScreenshot.id==None) \
 				.all()
@@ -68,56 +92,102 @@ def admin_page():
 				importRepoScreenshot.delay(package.id)
 
 			return redirect(url_for("admin.admin_page"))
+
 		elif action == "restore":
 			package = Package.query.get(request.form["package"])
 			if package is None:
 				flash("Unknown package", "danger")
 			else:
-				package.soft_deleted = False
+				package.state = PackageState.READY_FOR_REVIEW
 				db.session.commit()
 				return redirect(url_for("admin.admin_page"))
-		elif action == "importdepends":
-			task = importAllDependencies.delay()
-			return redirect(url_for("tasks.check", id=task.id, r=url_for("admin.admin_page")))
-		elif action == "modprovides":
-			packages = Package.query.filter_by(type=PackageType.MOD).all()
-			mpackage_cache = {}
-			for p in packages:
-				if len(p.provides) == 0:
-					p.provides.append(MetaPackage.GetOrCreate(p.name, mpackage_cache))
 
-			db.session.commit()
-			return redirect(url_for("admin.admin_page"))
 		elif action == "recalcscores":
 			for p in Package.query.all():
-				p.setStartScore()
+				p.recalcScore()
 
 			db.session.commit()
 			return redirect(url_for("admin.admin_page"))
-		elif action == "vcsrelease":
-			for package in Package.query.filter(Package.repo.isnot(None)).all():
-				if package.releases.count() != 0:
-					continue
 
-				rel = PackageRelease()
-				rel.package  = package
-				rel.title    = datetime.date.today().isoformat()
-				rel.url      = ""
-				rel.task_id  = uuid()
-				rel.approved = True
-				db.session.add(rel)
-				db.session.commit()
+		elif action == "cleanuploads":
+			upload_dir = app.config['UPLOAD_DIR']
 
-				makeVCSRelease.apply_async((rel.id, "master"), task_id=rel.task_id)
+			(_, _, filenames) = next(os.walk(upload_dir))
+			existing_uploads = set(filenames)
 
-				msg = "{}: Release {} created".format(package.title, rel.title)
-				triggerNotif(package.author, current_user, msg, rel.getEditURL())
-				db.session.commit()
+			if len(existing_uploads) != 0:
+				def getURLsFromDB(column):
+					results = db.session.query(column).filter(column != None, column != "").all()
+					return set([os.path.basename(x[0]) for x in results])
+
+				release_urls = getURLsFromDB(PackageRelease.url)
+				screenshot_urls = getURLsFromDB(PackageScreenshot.url)
+
+				db_urls = release_urls.union(screenshot_urls)
+				unreachable = existing_uploads.difference(db_urls)
+
+				import sys
+				print("On Disk: ", existing_uploads, file=sys.stderr)
+				print("In DB: ", db_urls, file=sys.stderr)
+				print("Unreachable: ", unreachable, file=sys.stderr)
+
+				for filename in unreachable:
+					os.remove(os.path.join(upload_dir, filename))
+
+				flash("Deleted " + str(len(unreachable)) + " unreachable uploads", "success")
+			else:
+				flash("No downloads to create", "danger")
+
+			return redirect(url_for("admin.admin_page"))
+
+		elif action == "delmetapackages":
+			query = MetaPackage.query.filter(~MetaPackage.dependencies.any(), ~MetaPackage.packages.any())
+			count = query.count()
+			query.delete(synchronize_session=False)
+			db.session.commit()
+
+			flash("Deleted " + str(count) + " unused meta packages", "success")
+			return redirect(url_for("admin.admin_page"))
+
+		elif action == "delremovedpackages":
+			query = Package.query.filter_by(state=PackageState.DELETED)
+			count = query.count()
+			for pkg in query.all():
+				pkg.review_thread = None
+				db.session.delete(pkg)
+			db.session.commit()
+
+			flash("Deleted {} soft deleted packages packages".format(count), "success")
+			return redirect(url_for("admin.admin_page"))
+
+		elif action == "addupdateconfig":
+			added = 0
+			for pkg in Package.query.filter(Package.repo != None, Package.releases.any(), Package.update_config == None).all():
+				pkg.update_config = PackageUpdateConfig()
+				pkg.update_config.auto_created = True
+
+				release: PackageRelease = pkg.releases.first()
+				if release and release.commit_hash:
+					pkg.update_config.last_commit = release.commit_hash
+
+				db.session.add(pkg.update_config)
+				added += 1
+
+			db.session.commit()
+
+			flash("Added {} update configs".format(added), "success")
+			return redirect(url_for("admin.admin_page"))
+
+		elif action == "runupdateconfig":
+			check_for_updates.delay()
+
+			flash("Started update configs", "success")
+			return redirect(url_for("admin.admin_page"))
 
 		else:
 			flash("Unknown action: " + action, "danger")
 
-	deleted_packages = Package.query.filter_by(soft_deleted=True).all()
+	deleted_packages = Package.query.filter(Package.state==PackageState.DELETED).all()
 	return render_template("admin/list.html", deleted_packages=deleted_packages)
 
 class SwitchUserForm(FlaskForm):
@@ -129,11 +199,11 @@ class SwitchUserForm(FlaskForm):
 @rank_required(UserRank.ADMIN)
 def switch_user():
 	form = SwitchUserForm(formdata=request.form)
-	if request.method == "POST" and form.validate():
+	if form.validate_on_submit():
 		user = User.query.filter_by(username=form["username"].data).first()
 		if user is None:
 			flash("Unable to find user", "danger")
-		elif loginUser(user):
+		elif login_user(user):
 			return redirect(url_for("users.profile", username=current_user.username))
 		else:
 			flash("Unable to login as user", "danger")
@@ -141,3 +211,26 @@ def switch_user():
 
 	# Process GET or invalid POST
 	return render_template("admin/switch_user.html", form=form)
+
+
+class SendNotificationForm(FlaskForm):
+	title  = StringField("Title", [InputRequired(), Length(1, 300)])
+	url    = StringField("URL", [InputRequired(), Length(1, 100)], default="/")
+	submit = SubmitField("Send")
+
+
+@bp.route("/admin/send-notification/", methods=["GET", "POST"])
+@rank_required(UserRank.ADMIN)
+def send_bulk_notification():
+	form = SendNotificationForm(request.form)
+	if form.validate_on_submit():
+		addAuditLog(AuditSeverity.MODERATION, current_user,
+				"Sent bulk notification", None, None, form.title.data)
+
+		users = User.query.filter(User.rank >= UserRank.NEW_MEMBER).all()
+		addNotification(users, current_user, NotificationType.OTHER, form.title.data, form.url.data, None)
+		db.session.commit()
+
+		return redirect(url_for("admin.admin_page"))
+
+	return render_template("admin/send_bulk_notification.html", form=form)

app/blueprints/admin/audit.py

@@ -0,0 +1,46 @@
+# ContentDB
+# Copyright (C) 2020  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+from flask import render_template, request, abort
+from app.models import db, AuditLogEntry, UserRank, User
+from app.utils import rank_required, get_int_or_abort
+
+from . import bp
+
+
+@bp.route("/admin/audit/")
+@rank_required(UserRank.MODERATOR)
+def audit():
+	page = get_int_or_abort(request.args.get("page"), 1)
+	num = min(40, get_int_or_abort(request.args.get("n"), 100))
+
+	query = AuditLogEntry.query.order_by(db.desc(AuditLogEntry.created_at))
+
+	if "username" in request.args:
+		user = User.query.filter_by(username=request.args.get("username")).first()
+		if not user:
+			abort(404)
+		query = query.filter_by(causer=user)
+
+	pagination = query.paginate(page, num, True)
+	return render_template("admin/audit.html", log=pagination.items, pagination=pagination)
+
+
+@bp.route("/admin/audit/<int:id>/")
+@rank_required(UserRank.MODERATOR)
+def audit_view(id):
+	entry = AuditLogEntry.query.get(id)
+	return render_template("admin/audit_view.html", entry=entry)

app/blueprints/admin/email.py

@@ -0,0 +1,79 @@
+# ContentDB
+# Copyright (C) 2020  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_login import current_user
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+
+from app.utils.markdown import render_markdown
+from app.models import *
+from app.tasks.emails import send_user_email
+from app.utils import rank_required, addAuditLog
+from . import bp
+
+
+class SendEmailForm(FlaskForm):
+	subject = StringField("Subject", [InputRequired(), Length(1, 300)])
+	text    = TextAreaField("Message", [InputRequired()])
+	submit  = SubmitField("Send")
+
+
+@bp.route("/admin/send-email/", methods=["GET", "POST"])
+@rank_required(UserRank.ADMIN)
+def send_single_email():
+	username = request.args["username"]
+	user = User.query.filter_by(username=username).first()
+	if user is None:
+		abort(404)
+
+	next_url = url_for("users.profile", username=user.username)
+
+	if user.email is None:
+		flash("User has no email address!", "danger")
+		return redirect(next_url)
+
+	form = SendEmailForm(request.form)
+	if form.validate_on_submit():
+		addAuditLog(AuditSeverity.MODERATION, current_user,
+				"Sent email to {}".format(user.display_name), url_for("users.profile", username=username))
+
+		text = form.text.data
+		html = render_markdown(text)
+		task = send_user_email.delay(user.email, form.subject.data, text, html)
+		return redirect(url_for("tasks.check", id=task.id, r=next_url))
+
+	return render_template("admin/send_email.html", form=form, user=user)
+
+
+@bp.route("/admin/send-bulk-email/", methods=["GET", "POST"])
+@rank_required(UserRank.ADMIN)
+def send_bulk_email():
+	form = SendEmailForm(request.form)
+	if form.validate_on_submit():
+		addAuditLog(AuditSeverity.MODERATION, current_user,
+				"Sent bulk email", None, None, form.text.data)
+
+		text = form.text.data
+		html = render_markdown(text)
+		for user in User.query.filter(User.email != None).all():
+			send_user_email.delay(user.email, form.subject.data, text, html)
+
+		return redirect(url_for("admin.admin_page"))
+
+	return render_template("admin/send_bulk_email.html", form=form)

app/blueprints/admin/licenseseditor.py

@@ -1,28 +1,29 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
 from flask import *
-from flask_user import *
-from . import bp
-from app.models import *
 from flask_wtf import FlaskForm
 from wtforms import *
 from wtforms.validators import *
+
+from app.models import *
 from app.utils import rank_required
+from . import bp
+
 
 @bp.route("/licenses/")
 @rank_required(UserRank.MODERATOR)
@@ -47,7 +48,7 @@ def create_edit_license(name=None):
 	form = LicenseForm(formdata=request.form, obj=license)
 	if request.method == "GET" and license is None:
 		form.is_foss.data = True
-	elif request.method == "POST" and form.validate():
+	elif form.validate_on_submit():
 		if license is None:
 			license = License(form.name.data)
 			db.session.add(license)

app/blueprints/admin/tagseditor.py

@@ -1,42 +1,54 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
 from flask import *
-from flask_user import *
-from . import bp
-from app.models import *
+from flask_login import current_user, login_required
 from flask_wtf import FlaskForm
 from wtforms import *
 from wtforms.validators import *
-from app.utils import rank_required
+
+from app.models import *
+from . import bp
+
 
 @bp.route("/tags/")
-@rank_required(UserRank.MODERATOR)
+@login_required
 def tag_list():
-	return render_template("admin/tags/list.html", tags=Tag.query.order_by(db.asc(Tag.title)).all())
+	if not Permission.EDIT_TAGS.check(current_user):
+		abort(403)
+
+	query = Tag.query
+
+	if request.args.get("sort") == "views":
+		query = query.order_by(db.desc(Tag.views))
+	else:
+		query = query.order_by(db.asc(Tag.title))
+
+	return render_template("admin/tags/list.html", tags=query.all())
 
 class TagForm(FlaskForm):
-	title	 = StringField("Title", [InputRequired(), Length(3,100)])
-	name     = StringField("Name", [Optional(), Length(1, 20), Regexp("^[a-z0-9_]", 0, "Lower case letters (a-z), digits (0-9), and underscores (_) only")])
-	submit   = SubmitField("Save")
+	title	    = StringField("Title", [InputRequired(), Length(3,100)])
+	description = TextAreaField("Description", [Optional(), Length(0, 500)])
+	name        = StringField("Name", [Optional(), Length(1, 20), Regexp("^[a-z0-9_]", 0, "Lower case letters (a-z), digits (0-9), and underscores (_) only")])
+	submit      = SubmitField("Save")
 
 @bp.route("/tags/new/", methods=["GET", "POST"])
 @bp.route("/tags/<name>/edit/", methods=["GET", "POST"])
-@rank_required(UserRank.MODERATOR)
+@login_required
 def create_edit_tag(name=None):
 	tag = None
 	if name is not None:
@@ -44,14 +56,22 @@ def create_edit_tag(name=None):
 		if tag is None:
 			abort(404)
 
+	if not Permission.checkPerm(current_user, Permission.EDIT_TAGS if tag else Permission.CREATE_TAG):
+		abort(403)
+
 	form = TagForm(formdata=request.form, obj=tag)
-	if request.method == "POST" and form.validate():
+	if form.validate_on_submit():
 		if tag is None:
 			tag = Tag(form.title.data)
+			tag.description = form.description.data
 			db.session.add(tag)
 		else:
 			form.populate_obj(tag)
 		db.session.commit()
-		return redirect(url_for("admin.create_edit_tag", name=tag.name))
+
+		if Permission.EDIT_TAGS.check(current_user):
+			return redirect(url_for("admin.create_edit_tag", name=tag.name))
+		else:
+			return redirect(url_for("homepage.home"))
 
 	return render_template("admin/tags/edit.html", tag=tag, form=form)

app/blueprints/admin/versioneditor.py

@@ -1,28 +1,29 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
 from flask import *
-from flask_user import *
-from . import bp
-from app.models import *
 from flask_wtf import FlaskForm
 from wtforms import *
 from wtforms.validators import *
+
+from app.models import *
 from app.utils import rank_required
+from . import bp
+
 
 @bp.route("/versions/")
 @rank_required(UserRank.MODERATOR)
@@ -45,7 +46,7 @@ def create_edit_version(name=None):
 			abort(404)
 
 	form = VersionForm(formdata=request.form, obj=version)
-	if request.method == "POST" and form.validate():
+	if form.validate_on_submit():
 		if version is None:
 			version = MinetestRelease(form.name.data)
 			db.session.add(version)

app/blueprints/admin/warningseditor.py

@@ -0,0 +1,60 @@
+# ContentDB
+# Copyright (C) 2020  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+
+from app.models import *
+from app.utils import rank_required
+from . import bp
+
+
+@bp.route("/admin/warnings/")
+@rank_required(UserRank.ADMIN)
+def warning_list():
+	return render_template("admin/warnings/list.html", warnings=ContentWarning.query.order_by(db.asc(ContentWarning.title)).all())
+
+class WarningForm(FlaskForm):
+	title	    = StringField("Title", [InputRequired(), Length(3,100)])
+	description = TextAreaField("Description", [Optional(), Length(0, 500)])
+	name        = StringField("Name", [Optional(), Length(1, 20), Regexp("^[a-z0-9_]", 0, "Lower case letters (a-z), digits (0-9), and underscores (_) only")])
+	submit      = SubmitField("Save")
+
+@bp.route("/admin/warnings/new/", methods=["GET", "POST"])
+@bp.route("/admin/warnings/<name>/edit/", methods=["GET", "POST"])
+@rank_required(UserRank.ADMIN)
+def create_edit_warning(name=None):
+	warning = None
+	if name is not None:
+		warning = ContentWarning.query.filter_by(name=name).first()
+		if warning is None:
+			abort(404)
+
+	form = WarningForm(formdata=request.form, obj=warning)
+	if form.validate_on_submit():
+		if warning is None:
+			warning = ContentWarning(form.title.data, form.description.data)
+			db.session.add(warning)
+		else:
+			form.populate_obj(warning)
+		db.session.commit()
+
+		return redirect(url_for("admin.warning_list"))
+
+	return render_template("admin/warnings/edit.html", warning=warning, form=form)

app/blueprints/api/__init__.py

@@ -1,17 +1,17 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 from flask import Blueprint

app/blueprints/api/auth.py

@@ -1,23 +1,27 @@
-# Content DB
+# ContentDB
 # Copyright (C) 2019  rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
-from flask import request, make_response, jsonify, abort
-from app.models import APIToken
 from functools import wraps
 
+from flask import request, abort
+
+from app.models import APIToken
+from .support import error
+
+
 def is_api_authd(f):
 	@wraps(f)
 	def decorated_function(*args, **kwargs):
@@ -29,13 +33,13 @@ def is_api_authd(f):
 		elif value[0:7].lower() == "bearer ":
 			access_token = value[7:]
 			if len(access_token) < 10:
-				abort(400)
+				error(400, "API token is too short")
 
 			token = APIToken.query.filter_by(access_token=access_token).first()
 			if token is None:
-				abort(403)
+				error(403, "Unknown API token")
 		else:
-			abort(403)
+			abort(403, "Unsupported authentication method")
 
 		return f(token=token, *args, **kwargs)
 

app/blueprints/api/endpoints.py

@@ -1,39 +1,44 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21  rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
+from flask import request, jsonify, current_app, abort
+from flask_login import current_user, login_required
+from sqlalchemy.sql.expression import func
 
-from flask import *
-from flask_user import *
+from app import csrf
+from app.utils.markdown import render_markdown
+from app.models import Tag, PackageState, PackageType, Package, db, PackageRelease, Permission, ForumTopic, MinetestRelease, APIToken, PackageScreenshot, License, ContentWarning
+from app.querybuilder import QueryBuilder
+from app.utils import is_package_page
 from . import bp
 from .auth import is_api_authd
-from .support import error, handleCreateRelease
-from app import csrf
-from app.models import *
-from app.utils import is_package_page
-from app.markdown import render_markdown
-from app.querybuilder import QueryBuilder
+from .support import error, api_create_vcs_release, api_create_zip_release, api_create_screenshot, api_order_screenshots, api_edit_package
+
 
 @bp.route("/api/packages/")
 def packages():
 	qb    = QueryBuilder(request.args)
 	query = qb.buildPackageQuery()
-	ver   = qb.getMinetestVersion()
 
-	pkgs = [package.getAsDictionaryShort(current_app.config["BASE_URL"], version=ver) \
-			for package in query.all()]
+	if request.args.get("fmt") == "keys":
+		return jsonify([package.getAsDictionaryKey() for package in query.all()])
+
+	pkgs = qb.convertToDictionary(query.all())
+	if "engine_version" in request.args or "protocol_version" in request.args:
+		pkgs = [package for package in pkgs if package.get("release")]
 	return jsonify(pkgs)
 
 
@@ -43,25 +48,41 @@ def package(package):
 	return jsonify(package.getAsDictionary(current_app.config["BASE_URL"]))
 
 
-@bp.route("/api/packages/<author>/<name>/dependencies/")
+@bp.route("/api/packages/<author>/<name>/", methods=["PUT"])
+@csrf.exempt
 @is_package_page
-def package_dependencies(package):
+@is_api_authd
+def edit_package(token, package):
+	if not token:
+		error(401, "Authentication needed")
+
+	return api_edit_package(token, package, request.json)
+
+
+def resolve_package_deps(out, package, only_hard):
+	id = package.getId()
+	if id in out:
+		return
+
 	ret = []
+	out[id] = ret
 
 	for dep in package.dependencies:
-		name = None
-		fulfilled_by = None
+		if only_hard and dep.optional:
+			continue
 
 		if dep.package:
 			name = dep.package.name
-			fulfilled_by = [ dep.package.getAsDictionaryKey() ]
+			fulfilled_by = [ dep.package.getId() ]
+			resolve_package_deps(out, dep.package, only_hard)
 
 		elif dep.meta_package:
 			name = dep.meta_package.name
-			fulfilled_by = [ pkg.getAsDictionaryKey() for pkg in dep.meta_package.packages]
+			fulfilled_by = [ pkg.getId() for pkg in dep.meta_package.packages]
+			# TODO: resolve most likely candidate
 
 		else:
-			raise "Malformed dependency"
+			raise Exception("Malformed dependency")
 
 		ret.append({
 			"name": name,
@@ -69,14 +90,16 @@ def package_dependencies(package):
 			"packages": fulfilled_by
 		})
 
-	return jsonify(ret)
 
-
-@bp.route("/api/packages/<author>/<name>/releases/")
+@bp.route("/api/packages/<author>/<name>/dependencies/")
 @is_package_page
-def list_releases(package):
-	releases = package.releases.filter_by(approved=True).all()
-	return jsonify([ rel.getAsDictionary() for rel in releases ])
+def package_dependencies(package):
+	only_hard = request.args.get("only_hard")
+
+	out = {}
+	resolve_package_deps(out, package, only_hard)
+
+	return jsonify(out)
 
 
 @bp.route("/api/topics/")
@@ -104,12 +127,6 @@ def topic_set_discard():
 	return jsonify(topic.getAsDictionary())
 
 
-@bp.route("/api/minetest_versions/")
-def versions():
-	return jsonify([{ "name": rel.name, "protocol_version": rel.protocol }\
-			for rel in MinetestRelease.query.all() if rel.getActual() is not None])
-
-
 @bp.route("/api/whoami/")
 @is_api_authd
 def whoami(token):
@@ -125,21 +142,229 @@ def markdown():
 	return render_markdown(request.data.decode("utf-8"))
 
 
+@bp.route("/api/packages/<author>/<name>/releases/")
+@is_package_page
+def list_releases(package):
+	return jsonify([ rel.getAsDictionary() for rel in package.releases.all() ])
+
+
 @bp.route("/api/packages/<author>/<name>/releases/new/", methods=["POST"])
 @csrf.exempt
 @is_package_page
 @is_api_authd
 def create_release(token, package):
+	if not token:
+		error(401, "Authentication needed")
+
+	if not package.checkPerm(token.owner, Permission.APPROVE_RELEASE):
+		error(403, "You do not have the permission to approve releases")
+
+	data = request.json or request.form
+	if "title" not in data:
+		error(400, "Title is required in the POST data")
+
+	if data.get("method") == "git":
+		for option in ["method", "ref"]:
+			if option not in data:
+				error(400, option + " is required in the POST data")
+
+		return api_create_vcs_release(token, package, data["title"], data["ref"])
+
+	elif request.files:
+		file = request.files.get("file")
+		if file is None:
+			error(400, "Missing 'file' in multipart body")
+
+		return api_create_zip_release(token, package, data["title"], file)
+
+	else:
+		error(400, "Unknown release-creation method. Specify the method or provide a file.")
+
+
+@bp.route("/api/packages/<author>/<name>/releases/<int:id>/")
+@is_package_page
+def release(package: Package, id: int):
+	release = PackageRelease.query.get(id)
+	if release is None or release.package != package:
+		error(404, "Release not found")
+
+	return jsonify(release.getAsDictionary())
+
+
+@bp.route("/api/packages/<author>/<name>/releases/<int:id>/", methods=["DELETE"])
+@csrf.exempt
+@is_package_page
+@is_api_authd
+def delete_release(token: APIToken, package: Package, id: int):
+	release = PackageRelease.query.get(id)
+	if release is None or release.package != package:
+		error(404, "Release not found")
+
+	if not token:
+		error(401, "Authentication needed")
+
+	if not token.canOperateOnPackage(package):
+		error(403, "API token does not have access to the package")
+
+	if not release.checkPerm(token.owner, Permission.DELETE_RELEASE):
+		error(403, "Unable to delete the release, make sure there's a newer release available")
+
+	db.session.delete(release)
+	db.session.commit()
+
+	return jsonify({"success": True})
+
+
+@bp.route("/api/packages/<author>/<name>/screenshots/")
+@is_package_page
+def list_screenshots(package):
+	screenshots = package.screenshots.all()
+	return jsonify([ss.getAsDictionary(current_app.config["BASE_URL"]) for ss in screenshots])
+
+
+@bp.route("/api/packages/<author>/<name>/screenshots/new/", methods=["POST"])
+@csrf.exempt
+@is_package_page
+@is_api_authd
+def create_screenshot(token: APIToken, package: Package):
+	if not token:
+		error(401, "Authentication needed")
+
+	if not package.checkPerm(token.owner, Permission.ADD_SCREENSHOTS):
+		error(403, "You do not have the permission to create screenshots")
+
+	data = request.form
+	if "title" not in data:
+		error(400, "Title is required in the POST data")
+
+	file = request.files.get("file")
+	if file is None:
+		error(400, "Missing 'file' in multipart body")
+
+	return api_create_screenshot(token, package, data["title"], file)
+
+
+@bp.route("/api/packages/<author>/<name>/screenshots/<int:id>/")
+@is_package_page
+def screenshot(package, id):
+	ss = PackageScreenshot.query.get(id)
+	if ss is None or ss.package != package:
+		error(404, "Screenshot not found")
+
+	return jsonify(ss.getAsDictionary(current_app.config["BASE_URL"]))
+
+
+@bp.route("/api/packages/<author>/<name>/screenshots/<int:id>/", methods=["DELETE"])
+@csrf.exempt
+@is_package_page
+@is_api_authd
+def delete_screenshot(token: APIToken, package: Package, id: int):
+	ss = PackageScreenshot.query.get(id)
+	if ss is None or ss.package != package:
+		error(404, "Screenshot not found")
+
+	if not token:
+		error(401, "Authentication needed")
+
+	if not package.checkPerm(token.owner, Permission.ADD_SCREENSHOTS):
+		error(403, "You do not have the permission to delete screenshots")
+
+	if not token.canOperateOnPackage(package):
+		error(403, "API token does not have access to the package")
+
+	if package.cover_image == ss:
+		package.cover_image = None
+		db.session.merge(package)
+
+	db.session.delete(ss)
+	db.session.commit()
+
+	return jsonify({ "success": True })
+
+
+@bp.route("/api/packages/<author>/<name>/screenshots/order/", methods=["POST"])
+@csrf.exempt
+@is_package_page
+@is_api_authd
+def order_screenshots(token: APIToken, package: Package):
+	if not token:
+		error(401, "Authentication needed")
+
+	if not package.checkPerm(token.owner, Permission.ADD_SCREENSHOTS):
+		error(403, "You do not have the permission to delete screenshots")
+
+	if not token.canOperateOnPackage(package):
+		error(403, "API token does not have access to the package")
+
 	json = request.json
-	if json is None:
-		return error(400, "JSON post data is required")
+	if json is None or not isinstance(json, list):
+		error(400, "Expected order body to be array")
 
-	for option in ["method", "title", "ref"]:
-		if json.get(option) is None:
-			return error(400, option + " is required in the POST data")
+	return api_order_screenshots(token, package, request.json)
 
 
-	if json["method"].lower() != "git":
-		return error(400, "Release-creation methods other than git are not supported")
+@bp.route("/api/scores/")
+def package_scores():
+	qb    = QueryBuilder(request.args)
+	query = qb.buildPackageQuery()
 
-	return handleCreateRelease(token, package, json["title"], json["ref"])
+	pkgs = [package.getScoreDict() for package in query.all()]
+	return jsonify(pkgs)
+
+
+@bp.route("/api/tags/")
+def tags():
+	return jsonify([tag.getAsDictionary() for tag in Tag.query.all() ])
+
+
+@bp.route("/api/content_warnings/")
+def content_warnings():
+	return jsonify([warning.getAsDictionary() for warning in ContentWarning.query.all() ])
+
+
+@bp.route("/api/licenses/")
+def licenses():
+	return jsonify([ { "name": license.name, "is_foss": license.is_foss } \
+		for license in License.query.order_by(db.asc(License.name)).all() ])
+
+
+@bp.route("/api/homepage/")
+def homepage():
+	query   = Package.query.filter_by(state=PackageState.APPROVED)
+	count   = query.count()
+
+	new     = query.order_by(db.desc(Package.approved_at)).limit(4).all()
+	pop_mod = query.filter_by(type=PackageType.MOD).order_by(db.desc(Package.score)).limit(8).all()
+	pop_gam = query.filter_by(type=PackageType.GAME).order_by(db.desc(Package.score)).limit(8).all()
+	pop_txp = query.filter_by(type=PackageType.TXP).order_by(db.desc(Package.score)).limit(8).all()
+	high_reviewed = query.order_by(db.desc(Package.score - Package.score_downloads)) \
+			.filter(Package.reviews.any()).limit(4).all()
+
+	updated = db.session.query(Package).select_from(PackageRelease).join(Package) \
+			.filter_by(state=PackageState.APPROVED) \
+			.order_by(db.desc(PackageRelease.releaseDate)) \
+			.limit(20).all()
+	updated = updated[:4]
+
+	downloads_result = db.session.query(func.sum(Package.downloads)).one_or_none()
+	downloads = 0 if not downloads_result or not downloads_result[0] else downloads_result[0]
+
+	def mapPackages(packages):
+		return [pkg.getAsDictionaryKey() for pkg in packages]
+
+	return {
+		"count": count,
+		"downloads": downloads,
+		"new": mapPackages(new),
+		"updated": mapPackages(updated),
+		"pop_mod": mapPackages(pop_mod),
+		"pop_txp": mapPackages(pop_txp),
+		"pop_game": mapPackages(pop_gam),
+		"high_reviewed": mapPackages(high_reviewed)
+	}
+
+
+@bp.route("/api/minetest_versions/")
+def versions():
+	return jsonify([rel.getAsDictionary() \
+			for rel in MinetestRelease.query.all() if rel.getActual() is not None])

app/blueprints/api/support.py

@@ -1,40 +1,108 @@
-from app.models import PackageRelease, db, Permission
-from app.tasks.importtasks import makeVCSRelease
-from celery import uuid
-from flask import jsonify, make_response, url_for
-import datetime
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-def error(status, message):
-	return make_response(jsonify({ "success": False, "error": message }), status)
+from flask import jsonify, abort, make_response, url_for, current_app
+
+from app.logic.packages import do_edit_package
+from app.logic.releases import LogicError, do_create_vcs_release, do_create_zip_release
+from app.logic.screenshots import do_create_screenshot, do_order_screenshots
+from app.models import APIToken, Package, MinetestRelease, PackageScreenshot
 
 
-def handleCreateRelease(token, package, title, ref):
+def error(code: int, msg: str):
+	abort(make_response(jsonify({ "success": False, "error": msg }), code))
+
+# Catches LogicErrors and aborts with JSON error
+def guard(f):
+	def ret(*args, **kwargs):
+		try:
+			return f(*args, **kwargs)
+		except LogicError as e:
+			error(e.code, e.message)
+
+	return ret
+
+
+def api_create_vcs_release(token: APIToken, package: Package, title: str, ref: str,
+		min_v: MinetestRelease = None, max_v: MinetestRelease = None, reason="API"):
 	if not token.canOperateOnPackage(package):
-		return error(403, "API token does not have access to the package")
+		error(403, "API token does not have access to the package")
 
-	if not package.checkPerm(token.owner, Permission.MAKE_RELEASE):
-		return error(403, "Permission denied. Missing MAKE_RELEASE permission")
+	reason += ", token=" + token.name
 
-	five_minutes_ago = datetime.datetime.now() - datetime.timedelta(minutes=5)
-	count = package.releases.filter(PackageRelease.releaseDate > five_minutes_ago).count()
-	if count >= 2:
-		return error(429, "Too many requests, please wait before trying again")
-
-	rel = PackageRelease()
-	rel.package = package
-	rel.title   = title
-	rel.url     = ""
-	rel.task_id = uuid()
-	rel.min_rel = None
-	rel.max_rel = None
-	db.session.add(rel)
-	db.session.commit()
-
-	makeVCSRelease.apply_async((rel.id, ref), task_id=rel.task_id)
+	rel = guard(do_create_vcs_release)(token.owner, package, title, ref, min_v, max_v, reason)
 
 	return jsonify({
 		"success": True,
 		"task": url_for("tasks.check", id=rel.task_id),
 		"release": rel.getAsDictionary()
 	})
+
+
+def api_create_zip_release(token: APIToken, package: Package, title: str, file,
+		min_v: MinetestRelease = None, max_v: MinetestRelease = None, reason="API"):
+	if not token.canOperateOnPackage(package):
+		error(403, "API token does not have access to the package")
+
+	reason += ", token=" + token.name
+
+	rel = guard(do_create_zip_release)(token.owner, package, title, file, min_v, max_v, reason)
+
+	return jsonify({
+		"success": True,
+		"task": url_for("tasks.check", id=rel.task_id),
+		"release": rel.getAsDictionary()
+	})
+
+
+def api_create_screenshot(token: APIToken, package: Package, title: str, file, reason="API"):
+	if not token.canOperateOnPackage(package):
+		error(403, "API token does not have access to the package")
+
+	reason += ", token=" + token.name
+
+	ss : PackageScreenshot = guard(do_create_screenshot)(token.owner, package, title, file, reason)
+
+	return jsonify({
+		"success": True,
+		"screenshot": ss.getAsDictionary()
+	})
+
+
+def api_order_screenshots(token: APIToken, package: Package, order: [any]):
+	if not token.canOperateOnPackage(package):
+		error(403, "API token does not have access to the package")
+
+	guard(do_order_screenshots)(token.owner, package, order)
+
+	return jsonify({
+		"success": True
+	})
+
+
+def api_edit_package(token: APIToken, package: Package, data: dict, reason: str = "API"):
+	if not token.canOperateOnPackage(package):
+		error(403, "API token does not have access to the package")
+
+	reason += ", token=" + token.name
+
+	package = guard(do_edit_package)(token.owner, package, False, data, reason)
+
+	return jsonify({
+		"success": True,
+		"package": package.getAsDictionary(current_app.config["BASE_URL"])
+	})

app/blueprints/api/tokens.py

@@ -1,39 +1,46 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
 from flask import render_template, redirect, request, session, url_for, abort
-from flask_user import login_required, current_user
-from . import bp
-from app.models import db, User, APIToken, Package, Permission
-from app.utils import randomString
-from app.querybuilder import QueryBuilder
-
+from flask_login import login_required, current_user
 from flask_wtf import FlaskForm
 from wtforms import *
-from wtforms.validators import *
 from wtforms.ext.sqlalchemy.fields import QuerySelectField
+from wtforms.validators import *
+
+from app.models import db, User, APIToken, Package, Permission
+from app.utils import randomString
+from . import bp
+from ..users.settings import get_setting_tabs
+
 
 class CreateAPIToken(FlaskForm):
 	name	     = StringField("Name", [InputRequired(), Length(1, 30)])
-	package      = QuerySelectField("Limit to package", allow_blank=True, \
+	package      = QuerySelectField("Limit to package", allow_blank=True,
 			get_pk=lambda a: a.id, get_label=lambda a: a.title)
 	submit	     = SubmitField("Save")
 
 
+@bp.route("/user/tokens/")
+@login_required
+def list_tokens_redirect():
+	return redirect(url_for("api.list_tokens", username=current_user.username))
+
+
 @bp.route("/users/<username>/tokens/")
 @login_required
 def list_tokens(username):
@@ -44,7 +51,7 @@ def list_tokens(username):
 	if not user.checkPerm(current_user, Permission.CREATE_TOKEN):
 		abort(403)
 
-	return render_template("api/list_tokens.html", user=user)
+	return render_template("api/list_tokens.html", user=user, tabs=get_setting_tabs(user), current_tab="api_tokens")
 
 
 @bp.route("/users/<username>/tokens/new/", methods=["GET", "POST"])
@@ -69,24 +76,25 @@ def create_edit_token(username, id=None):
 		elif token.owner != user:
 			abort(403)
 
-		access_token = session.pop("token_" + str(id), None)
+		access_token = session.pop("token_" + str(token.id), None)
 
 	form = CreateAPIToken(formdata=request.form, obj=token)
-	form.package.query_factory = lambda: Package.query.filter_by(author=user).all()
+	form.package.query_factory = lambda: user.maintained_packages.all()
 
-	if request.method == "POST" and form.validate():
+	if form.validate_on_submit():
 		if is_new:
 			token = APIToken()
 			token.owner = user
 			token.access_token = randomString(32)
 
-			# Store token so it can be shown in the edit page
-			session["token_" + str(token.id)] = token.access_token
-
 		form.populate_obj(token)
 		db.session.add(token)
 		db.session.commit() # save
 
+		if is_new:
+			# Store token so it can be shown in the edit page
+			session["token_" + str(token.id)] = token.access_token
+
 		return redirect(url_for("api.create_edit_token", username=username, id=token.id))
 
 	return render_template("api/create_edit_token.html", user=user, form=form, token=token, access_token=access_token)
@@ -102,8 +110,6 @@ def reset_token(username, id):
 	if not user.checkPerm(current_user, Permission.CREATE_TOKEN):
 		abort(403)
 
-	is_new = id is None
-
 	token = APIToken.query.get(id)
 	if token is None:
 		abort(404)

app/blueprints/github/__init__.py

@@ -1,35 +1,31 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 from flask import Blueprint
 
 bp = Blueprint("github", __name__)
 
-from flask import redirect, url_for, request, flash, abort, render_template, jsonify, current_app
-from flask_user import current_user, login_required
-from sqlalchemy import func
-from flask_github import GitHub
+from flask import redirect, url_for, request, flash, jsonify, current_app
+from flask_login import current_user
+from sqlalchemy import func, or_, and_
 from app import github, csrf
-from app.models import db, User, APIToken, Package, Permission
-from app.utils import loginUser, randomString, abs_url_for
-from app.blueprints.api.support import error, handleCreateRelease
-import hmac, requests, json
-
-from flask_wtf import FlaskForm
-from wtforms import SelectField, SubmitField
+from app.models import db, User, APIToken, Package, Permission, AuditSeverity
+from app.utils import abs_url_for, addAuditLog, login_user_set_active
+from app.blueprints.api.support import error, api_create_vcs_release
+import hmac, requests
 
 @bp.route("/github/start/")
 def start():
@@ -47,7 +43,7 @@ def callback(oauth_token):
 	next_url = request.args.get("next")
 	if oauth_token is None:
 		flash("Authorization failed [err=gh-oauth-login-failed]", "danger")
-		return redirect(url_for("user.login"))
+		return redirect(url_for("users.login"))
 
 	# Get Github username
 	url = "https://api.github.com/user"
@@ -72,15 +68,19 @@ def callback(oauth_token):
 	else:
 		if userByGithub is None:
 			flash("Unable to find an account for that Github user", "danger")
-			return redirect(url_for("users.claim"))
-		elif loginUser(userByGithub):
-			if not current_user.hasPassword():
+			return redirect(url_for("users.claim_forums"))
+		elif login_user_set_active(userByGithub, remember=True):
+			addAuditLog(AuditSeverity.USER, userByGithub, "Logged in using GitHub OAuth",
+					url_for("users.profile", username=userByGithub.username))
+			db.session.commit()
+
+			if not current_user.password:
 				return redirect(next_url or url_for("users.set_password", optional=True))
 			else:
 				return redirect(next_url or url_for("homepage.home"))
 		else:
 			flash("Authorization failed [err=gh-login-failed]", "danger")
-			return redirect(url_for("user.login"))
+			return redirect(url_for("users.login"))
 
 
 @bp.route("/github/webhook/", methods=["POST"])
@@ -90,12 +90,15 @@ def webhook():
 
 	# Get package
 	github_url = "github.com/" + json["repository"]["full_name"]
-	package = Package.query.filter(Package.repo.like("%{}%".format(github_url))).first()
+	package = Package.query.filter(Package.repo.ilike("%{}%".format(github_url))).first()
 	if package is None:
-		return error(400, "Unknown package")
+		return error(400, "Could not find package, did you set the VCS repo in CDB correctly? Expected {}".format(github_url))
 
 	# Get all tokens for package
-	possible_tokens = APIToken.query.filter_by(package=package).all()
+	tokens_query = APIToken.query.filter(or_(APIToken.package==package,
+			and_(APIToken.package==None, APIToken.owner==package.author)))
+
+	possible_tokens = tokens_query.all()
 	actual_token = None
 
 	#
@@ -118,10 +121,10 @@ def webhook():
 			break
 
 	if actual_token is None:
-		return error(403, "Invalid authentication")
+		return error(403, "Invalid authentication, couldn't validate API token")
 
 	if not package.checkPerm(actual_token.owner, Permission.APPROVE_RELEASE):
-		return error(403, "Only trusted members can use webhooks")
+		return error(403, "You do not have the permission to approve releases")
 
 	#
 	# Check event
@@ -143,128 +146,7 @@ def webhook():
 	# Perform release
 	#
 
-	return handleCreateRelease(actual_token, package, title, ref)
+	if package.releases.filter_by(commit_hash=ref).count() > 0:
+		return
 
-
-class SetupWebhookForm(FlaskForm):
-	event   = SelectField("Event Type", choices=[('create', 'New tag or GitHub release'), ('push', 'Push')])
-	submit  = SubmitField("Save")
-
-
-@bp.route("/github/callback/webhook/")
-@github.authorized_handler
-def callback_webhook(oauth_token=None):
-	pid = request.args.get("pid")
-	if pid is None:
-		abort(404)
-
-	current_user.github_access_token = oauth_token
-	db.session.commit()
-
-	return redirect(url_for("github.setup_webhook", pid=pid))
-
-
-@bp.route("/github/webhook/new/", methods=["GET", "POST"])
-@login_required
-def setup_webhook():
-	pid = request.args.get("pid")
-	if pid is None:
-		abort(404)
-
-	package = Package.query.get(pid)
-	if package is None:
-		abort(404)
-
-	if not package.checkPerm(current_user, Permission.APPROVE_RELEASE):
-		flash("Only trusted members can use webhooks", "danger")
-		return redirect(package.getDetailsURL())
-
-	gh_user, gh_repo = package.getGitHubFullName()
-	if gh_user is None or gh_repo is None:
-		flash("Unable to get Github full name from repo address", "danger")
-		return redirect(package.getDetailsURL())
-
-	if current_user.github_access_token is None:
-		return github.authorize("write:repo_hook", \
-			redirect_uri=abs_url_for("github.callback_webhook", pid=pid))
-
-	form = SetupWebhookForm(formdata=request.form)
-	if request.method == "POST" and form.validate():
-		token = APIToken()
-		token.name = "GitHub Webhook for " + package.title
-		token.owner = current_user
-		token.access_token = randomString(32)
-		token.package = package
-
-		event = form.event.data
-		if event != "push" and event != "create":
-			abort(500)
-
-		if handleMakeWebhook(gh_user, gh_repo, package, \
-				current_user.github_access_token, event, token):
-			flash("Successfully created webhook", "success")
-			return redirect(package.getDetailsURL())
-		else:
-			return redirect(url_for("github.setup_webhook", pid=package.id))
-
-	return render_template("github/setup_webhook.html", \
-		form=form, package=package)
-
-
-def handleMakeWebhook(gh_user, gh_repo, package, oauth, event, token):
-	url = "https://api.github.com/repos/{}/{}/hooks".format(gh_user, gh_repo)
-	headers = {
-		"Authorization": "token " + oauth
-	}
-	data = {
-		"name": "web",
-		"active": True,
-		"events": [event],
-		"config": {
-			"url": abs_url_for("github.webhook"),
-			"content_type": "json",
-			"secret": token.access_token
-		},
-	}
-
-	# First check that the webhook doesn't already exist
-	r = requests.get(url, headers=headers)
-
-	if r.status_code == 401 or r.status_code == 403:
-		current_user.github_access_token = None
-		db.session.commit()
-		return False
-
-	if r.status_code != 200:
-		flash("Failed to create webhook, received response from Github " +
-			str(r.status_code) + ": " +
-			str(r.json().get("message")), "danger")
-		return False
-
-	for hook in r.json():
-		if hook.get("config") and hook["config"].get("url") and \
-				hook["config"]["url"] == data["config"]["url"]:
-			flash("Failed to create webhook, as it already exists", "danger")
-			return False
-
-
-	# Create it
-	r = requests.post(url, headers=headers, data=json.dumps(data))
-
-	if r.status_code == 201:
-		db.session.add(token)
-		db.session.commit()
-
-		return True
-
-	elif r.status_code == 401 or r.status_code == 403:
-		current_user.github_access_token = None
-		db.session.commit()
-
-		return False
-
-	else:
-		flash("Failed to create webhook, received response from Github " +
-			str(r.status_code) + ": " +
-			str(r.json().get("message")), "danger")
-		return False
+	return api_create_vcs_release(actual_token, package, title, ref, reason="Webhook")

app/blueprints/gitlab/__init__.py

@@ -2,16 +2,16 @@
 # Copyright (C) 2020  rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 from flask import Blueprint, request
@@ -20,19 +20,18 @@ bp = Blueprint("gitlab", __name__)
 
 from app import csrf
 from app.models import Package, APIToken, Permission
-from app.blueprints.api.support import error, handleCreateRelease
+from app.blueprints.api.support import error, api_create_vcs_release
 
 
-@bp.route("/gitlab/webhook/", methods=["POST"])
-@csrf.exempt
-def webhook():
+def webhook_impl():
 	json = request.json
 
 	# Get package
-	gitlab_url = "gitlab.com/{}/{}".format(json["project"]["namespace"], json["project"]["name"])
-	package = Package.query.filter(Package.repo.like("%{}%".format(gitlab_url))).first()
+	gitlab_url = json["project"]["web_url"].replace("https://", "").replace("http://", "")
+	package = Package.query.filter(Package.repo.ilike("%{}%".format(gitlab_url))).first()
 	if package is None:
-		return error(400, "Unknown package")
+		return error(400,
+				"Could not find package, did you set the VCS repo in CDB correctly? Expected {}".format(gitlab_url))
 
 	# Get all tokens for package
 	secret = request.headers.get("X-Gitlab-Token")
@@ -40,11 +39,11 @@ def webhook():
 		return error(403, "Token required")
 
 	token = APIToken.query.filter_by(access_token=secret).first()
-	if secret is None:
+	if token is None:
 		return error(403, "Invalid authentication")
 
 	if not package.checkPerm(token.owner, Permission.APPROVE_RELEASE):
-		return error(403, "Only trusted members can use webhooks")
+		return error(403, "You do not have the permission to approve releases")
 
 	#
 	# Check event
@@ -64,4 +63,16 @@ def webhook():
 	# Perform release
 	#
 
-	return handleCreateRelease(token, package, title, ref)
+	if package.releases.filter_by(commit_hash=ref).count() > 0:
+		return
+
+	return api_create_vcs_release(token, package, title, ref, reason="Webhook")
+
+
+@bp.route("/gitlab/webhook/", methods=["POST"])
+@csrf.exempt
+def webhook():
+	try:
+		return webhook_impl()
+	except KeyError as err:
+		return error(400, "Missing field: {}".format(err.args[0]))

app/blueprints/homepage/__init__.py

@@ -4,18 +4,40 @@ bp = Blueprint("homepage", __name__)
 
 from app.models import *
 import flask_menu as menu
+from sqlalchemy.orm import joinedload
 from sqlalchemy.sql.expression import func
 
 @bp.route("/")
 @menu.register_menu(bp, ".", "Home")
 def home():
-	query   = Package.query.filter_by(approved=True, soft_deleted=False)
+	def join(query):
+		return query.options(
+				joinedload(Package.license),
+				joinedload(Package.media_license))
+
+	query   = Package.query.filter_by(state=PackageState.APPROVED)
 	count   = query.count()
-	new     = query.order_by(db.desc(Package.created_at)).limit(8).all()
-	pop_mod = query.filter_by(type=PackageType.MOD).order_by(db.desc(Package.score)).limit(8).all()
-	pop_gam = query.filter_by(type=PackageType.GAME).order_by(db.desc(Package.score)).limit(4).all()
-	pop_txp = query.filter_by(type=PackageType.TXP).order_by(db.desc(Package.score)).limit(4).all()
-	downloads_result = db.session.query(func.sum(PackageRelease.downloads)).one_or_none()
+
+	new     = join(query.order_by(db.desc(Package.approved_at))).limit(4).all()
+	pop_mod = join(query.filter_by(type=PackageType.MOD).order_by(db.desc(Package.score))).limit(8).all()
+	pop_gam = join(query.filter_by(type=PackageType.GAME).order_by(db.desc(Package.score))).limit(8).all()
+	pop_txp = join(query.filter_by(type=PackageType.TXP).order_by(db.desc(Package.score))).limit(8).all()
+	high_reviewed = join(query.order_by(db.desc(Package.score - Package.score_downloads))) \
+			.filter(Package.reviews.any()).limit(4).all()
+
+	updated = db.session.query(Package).select_from(PackageRelease).join(Package) \
+			.filter_by(state=PackageState.APPROVED) \
+			.order_by(db.desc(PackageRelease.releaseDate)) \
+			.limit(20).all()
+	updated = updated[:4]
+
+	reviews = PackageReview.query.filter_by(recommends=True).order_by(db.desc(PackageReview.created_at)).limit(5).all()
+
+	downloads_result = db.session.query(func.sum(Package.downloads)).one_or_none()
 	downloads = 0 if not downloads_result or not downloads_result[0] else downloads_result[0]
-	return render_template("index.html", count=count, downloads=downloads, \
-			new=new, pop_mod=pop_mod, pop_txp=pop_txp, pop_gam=pop_gam)
+
+	tags = db.session.query(func.count(Tags.c.tag_id), Tag) \
+		.select_from(Tag).outerjoin(Tags).group_by(Tag.id).order_by(db.asc(Tag.title)).all()
+
+	return render_template("index.html", count=count, downloads=downloads, tags=tags,
+			new=new, updated=updated, pop_mod=pop_mod, pop_txp=pop_txp, pop_gam=pop_gam, high_reviewed=high_reviewed, reviews=reviews)

app/blueprints/metapackages/__init__.py

@@ -1,31 +1,35 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
 from flask import *
+from sqlalchemy import func
+from app.models import MetaPackage, Package, db, Dependency, PackageState, ForumTopic
 
 bp = Blueprint("metapackages", __name__)
 
-from flask_user import *
-from app.models import *
 
 @bp.route("/metapackages/")
 def list_all():
-	mpackages = MetaPackage.query.order_by(db.asc(MetaPackage.name)).all()
-	return render_template("meta/list.html", mpackages=mpackages)
+	mpackages = db.session.query(MetaPackage, func.count(Package.id)) \
+			.select_from(MetaPackage).outerjoin(MetaPackage.packages) \
+			.order_by(db.asc(MetaPackage.name)) \
+			.group_by(MetaPackage.id).all()
+	return render_template("metapackages/list.html", mpackages=mpackages)
+
 
 @bp.route("/metapackages/<name>/")
 def view(name):
@@ -33,4 +37,29 @@ def view(name):
 	if mpackage is None:
 		abort(404)
 
-	return render_template("meta/view.html", mpackage=mpackage)
+	dependers = db.session.query(Package) \
+		.select_from(MetaPackage) \
+		.filter(MetaPackage.name==name) \
+		.join(MetaPackage.dependencies) \
+		.join(Dependency.depender) \
+		.filter(Dependency.optional==False, Package.state==PackageState.APPROVED) \
+		.all()
+
+	optional_dependers = db.session.query(Package) \
+		.select_from(MetaPackage) \
+		.filter(MetaPackage.name==name) \
+		.join(MetaPackage.dependencies) \
+		.join(Dependency.depender) \
+		.filter(Dependency.optional==True, Package.state==PackageState.APPROVED) \
+		.all()
+
+	similar_topics = None
+	if mpackage.packages.filter_by(state=PackageState.APPROVED).count() == 0:
+		similar_topics = ForumTopic.query \
+				.filter_by(name=name) \
+				.order_by(db.asc(ForumTopic.name), db.asc(ForumTopic.title)) \
+				.all()
+
+	return render_template("metapackages/view.html", mpackage=mpackage,
+			dependers=dependers, optional_dependers=optional_dependers,
+			similar_topics=similar_topics)

app/blueprints/metrics/__init__.py

@@ -0,0 +1,74 @@
+# ContentDB
+# Copyright (C) 2020  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+from flask import Blueprint, make_response
+from sqlalchemy.sql.expression import func
+
+from app.models import Package, db, User, UserRank, PackageState
+
+bp = Blueprint("metrics", __name__)
+
+def generate_metrics(full=False):
+	def write_single_stat(name, help, type, value):
+		fmt = "# HELP {name} {help}\n# TYPE {name} {type}\n{name} {value}\n\n"
+
+		return fmt.format(name=name, help=help, type=type, value=value)
+
+	def gen_labels(labels):
+		pieces = [key + "=" + str(val) for key, val in labels.items()]
+		return ",".join(pieces)
+
+
+	def write_array_stat(name, help, type, data):
+		ret = "# HELP {name} {help}\n# TYPE {name} {type}\n" \
+			.format(name=name, help=help, type=type)
+
+		for entry in data:
+			assert(len(entry) == 2)
+			ret += "{name}{{{labels}}} {value}\n" \
+				.format(name=name, labels=gen_labels(entry[0]), value=entry[1])
+
+		return ret + "\n"
+
+	downloads_result = db.session.query(func.sum(Package.downloads)).one_or_none()
+	downloads = 0 if not downloads_result or not downloads_result[0] else downloads_result[0]
+
+	packages = Package.query.filter_by(state=PackageState.APPROVED).count()
+	users = User.query.filter(User.rank != UserRank.NOT_JOINED).count()
+
+	ret = ""
+	ret += write_single_stat("contentdb_packages", "Total packages", "counter", packages)
+	ret += write_single_stat("contentdb_users", "Number of registered users", "counter", users)
+	ret += write_single_stat("contentdb_downloads", "Total downloads", "counter", downloads)
+
+	if full:
+		scores = Package.query.join(User).with_entities(User.username, Package.name, Package.score) \
+			.filter(Package.state==PackageState.APPROVED).all()
+
+		ret += write_array_stat("contentdb_package_score", "Package score", "gauge",
+				[({ "author": score[0], "name": score[1] }, score[2])  for score in scores])
+	else:
+		score_result = db.session.query(func.sum(Package.score)).one_or_none()
+		score = 0 if not score_result or not score_result[0] else score_result[0]
+		ret += write_single_stat("contentdb_score", "Total package score", "gauge", score)
+
+	return ret
+
+@bp.route("/metrics")
+def metrics():
+	response = make_response(generate_metrics(), 200)
+	response.mimetype = "text/plain"
+	return response

app/blueprints/notifications/__init__.py

@@ -1,34 +1,49 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
 from flask import Blueprint, render_template, redirect, url_for
-from flask_user import current_user, login_required
-from app.models import db
+from flask_login import current_user, login_required
+from sqlalchemy import or_, desc
+
+from app.models import db, Notification, NotificationType
 
 bp = Blueprint("notifications", __name__)
 
+
 @bp.route("/notifications/")
 @login_required
 def list_all():
-	return render_template("notifications/list.html")
+	notifications = Notification.query.filter(Notification.user == current_user,
+			Notification.type != NotificationType.EDITOR_ALERT, Notification.type != NotificationType.EDITOR_MISC) \
+			.order_by(desc(Notification.created_at)) \
+			.all()
+
+	editor_notifications = Notification.query.filter(Notification.user == current_user,
+			or_(Notification.type == NotificationType.EDITOR_ALERT, Notification.type == NotificationType.EDITOR_MISC)) \
+			.order_by(desc(Notification.created_at)) \
+			.all()
+
+	return render_template("notifications/list.html",
+			notifications=notifications, editor_notifications=editor_notifications)
+
 
 @bp.route("/notifications/clear/", methods=["POST"])
 @login_required
 def clear():
-	current_user.notifications.clear()
+	Notification.query.filter_by(user=current_user).delete()
 	db.session.commit()
 	return redirect(url_for("notifications.list_all"))

app/blueprints/packages/__init__.py

@@ -1,21 +1,21 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 from flask import Blueprint
 
 bp = Blueprint("packages", __name__)
 
-from . import packages, screenshots, releases
+from . import packages, screenshots, releases, reviews

app/blueprints/packages/editrequests.py

@@ -1,173 +0,0 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-from flask import *
-from flask_user import *
-from app import app
-from app.models import *
-
-from app.utils import *
-
-from flask_wtf import FlaskForm
-from wtforms import *
-from wtforms.validators import *
-from wtforms.ext.sqlalchemy.fields import QuerySelectField, QuerySelectMultipleField
-
-from . import PackageForm
-
-
-class EditRequestForm(PackageForm):
-	edit_title = StringField("Edit Title", [InputRequired(), Length(1, 100)])
-	edit_desc  = TextField("Edit Description", [Optional()])
-
-@app.route("/packages/<author>/<name>/requests/new/", methods=["GET","POST"])
-@app.route("/packages/<author>/<name>/requests/<id>/edit/", methods=["GET","POST"])
-@login_required
-@is_package_page
-def create_edit_editrequest_page(package, id=None):
-	edited_package = package
-
-	erequest = None
-	if id is not None:
-		erequest = EditRequest.query.get(id)
-		if erequest.package != package:
-			abort(404)
-
-		if not erequest.checkPerm(current_user, Permission.EDIT_EDITREQUEST):
-			abort(403)
-
-		if erequest.status != 0:
-			flash("Can't edit EditRequest, it has already been merged or rejected", "danger")
-			return redirect(erequest.getURL())
-
-		edited_package = Package(package)
-		erequest.applyAll(edited_package)
-
-	form = EditRequestForm(request.form, obj=edited_package)
-	if request.method == "GET":
-		deps = edited_package.dependencies
-		form.harddep_str.data  = ",".join([str(x) for x in deps if not x.optional])
-		form.softdep_str.data  = ",".join([str(x) for x in deps if     x.optional])
-		form.provides_str.data = MetaPackage.ListToSpec(edited_package.provides)
-
-	if request.method == "POST" and form.validate():
-		if erequest is None:
-			erequest = EditRequest()
-			erequest.package = package
-			erequest.author  = current_user
-
-		erequest.title   = form["edit_title"].data
-		erequest.desc    = form["edit_desc"].data
-		db.session.add(erequest)
-
-		EditRequestChange.query.filter_by(request=erequest).delete()
-
-		wasChangeMade = False
-		for e in PackagePropertyKey:
-			newValue = form[e.name].data
-			oldValue = getattr(package, e.name)
-
-			newValueComp = newValue
-			oldValueComp = oldValue
-			if type(newValue) is str:
-				newValue = newValue.replace("\r\n", "\n")
-				newValueComp = newValue.strip()
-				oldValueComp = "" if oldValue is None else oldValue.strip()
-
-			if newValueComp != oldValueComp:
-				change = EditRequestChange()
-				change.request = erequest
-				change.key = e
-				change.oldValue = e.convert(oldValue)
-				change.newValue = e.convert(newValue)
-				db.session.add(change)
-				wasChangeMade = True
-
-		if wasChangeMade:
-			msg = "{}: Edit request #{} {}" \
-					.format(package.title, erequest.id, "created" if id is None else "edited")
-			triggerNotif(package.author, current_user, msg, erequest.getURL())
-			triggerNotif(erequest.author, current_user, msg, erequest.getURL())
-			db.session.commit()
-			return redirect(erequest.getURL())
-		else:
-			flash("No changes detected", "warning")
-	elif erequest is not None:
-		form["edit_title"].data = erequest.title
-		form["edit_desc"].data  = erequest.desc
-
-	return render_template("packages/editrequest_create_edit.html", package=package, form=form)
-
-
-@app.route("/packages/<author>/<name>/requests/<id>/")
-@is_package_page
-def view_editrequest_page(package, id):
-	erequest = EditRequest.query.get(id)
-	if erequest is None or erequest.package != package:
-		abort(404)
-
-	clearNotifications(erequest.getURL())
-	return render_template("packages/editrequest_view.html", package=package, request=erequest)
-
-
-@app.route("/packages/<author>/<name>/requests/<id>/approve/", methods=["POST"])
-@is_package_page
-def approve_editrequest_page(package, id):
-	if not package.checkPerm(current_user, Permission.APPROVE_CHANGES):
-		flash("You don't have permission to do that.", "danger")
-		return redirect(package.getDetailsURL())
-
-	erequest = EditRequest.query.get(id)
-	if erequest is None or erequest.package != package:
-		abort(404)
-
-	if erequest.status != 0:
-		flash("Edit request has already been resolved", "danger")
-
-	else:
-		erequest.status = 1
-		erequest.applyAll(package)
-
-		msg = "{}: Edit request #{} merged".format(package.title, erequest.id)
-		triggerNotif(erequest.author, current_user, msg, erequest.getURL())
-		triggerNotif(package.author, current_user, msg, erequest.getURL())
-		db.session.commit()
-
-	return redirect(package.getDetailsURL())
-
-@app.route("/packages/<author>/<name>/requests/<id>/reject/", methods=["POST"])
-@is_package_page
-def reject_editrequest_page(package, id):
-	if not package.checkPerm(current_user, Permission.APPROVE_CHANGES):
-		flash("You don't have permission to do that.", "danger")
-		return redirect(package.getDetailsURL())
-
-	erequest = EditRequest.query.get(id)
-	if erequest is None or erequest.package != package:
-		abort(404)
-
-	if erequest.status != 0:
-		flash("Edit request has already been resolved", "danger")
-
-	else:
-		erequest.status = 2
-
-		msg = "{}: Edit request #{} rejected".format(package.title, erequest.id)
-		triggerNotif(erequest.author, current_user, msg, erequest.getURL())
-		triggerNotif(package.author, current_user, msg, erequest.getURL())
-		db.session.commit()
-
-	return redirect(package.getDetailsURL())

app/blueprints/packages/packages.py

@@ -1,36 +1,40 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-from flask import render_template, abort, request, redirect, url_for, flash
-from flask_user import current_user
+from urllib.parse import quote as urlescape
+
 import flask_menu as menu
-
-from . import bp
-
-from app.models import *
-from app.querybuilder import QueryBuilder
-from app.tasks.importtasks import importRepoScreenshot
-from app.utils import *
-
+from celery import uuid
+from flask import render_template, flash
 from flask_wtf import FlaskForm
+from flask_login import login_required
+from sqlalchemy import or_, func
+from sqlalchemy.orm import joinedload, subqueryload
 from wtforms import *
-from wtforms.validators import *
 from wtforms.ext.sqlalchemy.fields import QuerySelectField, QuerySelectMultipleField
-from sqlalchemy import or_
+from wtforms.validators import *
+
+from app.querybuilder import QueryBuilder
+from app.rediscache import has_key, set_key
+from app.tasks.importtasks import importRepoScreenshot, checkZipRelease
+from app.utils import *
+from . import bp
+from ...logic.LogicError import LogicError
+from ...logic.packages import do_edit_package
 
 
 @menu.register_menu(bp, ".mods", "Mods", order=11, endpoint_arguments_constructor=lambda: { 'type': 'mod' })
@@ -43,6 +47,26 @@ def list_all():
 	query = qb.buildPackageQuery()
 	title = qb.title
 
+	query = query.options(
+			joinedload(Package.license),
+			joinedload(Package.media_license),
+			subqueryload(Package.tags))
+
+	ip = request.headers.get("X-Forwarded-For") or request.remote_addr
+	if ip is not None and not is_user_bot():
+		edited = False
+		for tag in qb.tags:
+			edited = True
+			key = "tag/{}/{}".format(ip, tag.name)
+			if not has_key(key):
+				set_key(key, "true")
+				Tag.query.filter_by(id=tag.id).update({
+						"views": Tag.views + 1
+					})
+
+		if edited:
+			db.session.commit()
+
 	if qb.lucky:
 		package = query.first()
 		if package:
@@ -59,21 +83,29 @@ def list_all():
 	search = request.args.get("q")
 	type_name = request.args.get("type")
 
-	next_url = url_for("packages.list_all", type=type_name, q=search, page=query.next_num) \
-			if query.has_next else None
-	prev_url = url_for("packages.list_all", type=type_name, q=search, page=query.prev_num) \
-			if query.has_prev else None
+	authors = []
+	if search:
+		authors = User.query \
+			.filter(or_(*[func.lower(User.username) == name.lower().strip() for name in search.split(" ")])) \
+			.all()
+
+		authors = [(author.username, search.lower().replace(author.username.lower(), "")) for author in authors]
 
 	topics = None
 	if qb.search and not query.has_next:
 		qb.show_discarded = True
 		topics = qb.buildTopicQuery().all()
 
-	tags = Tag.query.all()
-	return render_template("packages/list.html", \
-			title=title, packages=query.items, topics=topics, \
-			query=search, tags=tags, type=type_name, \
-			next_url=next_url, prev_url=prev_url, page=page, page_max=query.pages, packages_count=query.total)
+	tags_query = db.session.query(func.count(Tags.c.tag_id), Tag) \
+  		.select_from(Tag).join(Tags).join(Package).group_by(Tag.id).order_by(db.asc(Tag.title))
+	tags = qb.filterPackageQuery(tags_query).all()
+
+	selected_tags = set(qb.tags)
+
+	return render_template("packages/list.html",
+			title=title, packages=query.items, pagination=query,
+			query=search, tags=tags, selected_tags=selected_tags, type=type_name,
+			authors=authors, packages_count=query.total, topics=topics)
 
 
 def getReleases(package):
@@ -86,13 +118,11 @@ def getReleases(package):
 @bp.route("/packages/<author>/<name>/")
 @is_package_page
 def view(package):
-	clearNotifications(package.getDetailsURL())
-
 	alternatives = None
 	if package.type == PackageType.MOD:
 		alternatives = Package.query \
-			.filter_by(name=package.name, type=PackageType.MOD, soft_deleted=False) \
-			.filter(Package.id != package.id) \
+			.filter_by(name=package.name, type=PackageType.MOD) \
+			.filter(Package.id != package.id, Package.state!=PackageState.DELETED) \
 			.order_by(db.desc(Package.score)) \
 			.all()
 
@@ -109,7 +139,6 @@ def view(package):
 				.all()
 
 	releases = getReleases(package)
-	requests = [r for r in package.requests if r.status == 0]
 
 	review_thread = package.review_thread
 	if review_thread is not None and not review_thread.checkPerm(current_user, Permission.SEE_THREAD):
@@ -117,9 +146,9 @@ def view(package):
 
 	topic_error = None
 	topic_error_lvl = "warning"
-	if not package.approved and package.forums is not None:
+	if package.state != PackageState.APPROVED and package.forums is not None:
 		errors = []
-		if Package.query.filter_by(forums=package.forums, soft_deleted=False).count() > 1:
+		if Package.query.filter(Package.forums==package.forums, Package.state!=PackageState.DELETED).count() > 1:
 			errors.append("<b>Error: Another package already uses this forum topic!</b>")
 			topic_error_lvl = "danger"
 
@@ -128,27 +157,42 @@ def view(package):
 			if topic.author != package.author:
 				errors.append("<b>Error: Forum topic author doesn't match package author.</b>")
 				topic_error_lvl = "danger"
-
-			if topic.wip:
-				errors.append("Warning: Forum topic is in WIP section, make sure package meets playability standards.")
 		elif package.type != PackageType.TXP:
 			errors.append("Warning: Forum topic not found. This may happen if the topic has only just been created.")
 
 		topic_error = "<br />".join(errors)
 
 
-	threads = Thread.query.filter_by(package_id=package.id)
+	threads = Thread.query.filter_by(package_id=package.id, review_id=None)
 	if not current_user.is_authenticated:
 		threads = threads.filter_by(private=False)
 	elif not current_user.rank.atLeast(UserRank.EDITOR) and not current_user == package.author:
 		threads = threads.filter(or_(Thread.private == False, Thread.author == current_user))
 
+	has_review = current_user.is_authenticated and PackageReview.query.filter_by(package=package, author=current_user).count() > 0
 
-	return render_template("packages/view.html", \
-			package=package, releases=releases, requests=requests, \
-			alternatives=alternatives, similar_topics=similar_topics, \
-			review_thread=review_thread, topic_error=topic_error, topic_error_lvl=topic_error_lvl, \
-			threads=threads.all())
+	return render_template("packages/view.html",
+			package=package, releases=releases,
+			alternatives=alternatives, similar_topics=similar_topics,
+			review_thread=review_thread, topic_error=topic_error, topic_error_lvl=topic_error_lvl,
+			threads=threads.all(), has_review=has_review)
+
+
+@bp.route("/packages/<author>/<name>/shields/<type>/")
+@is_package_page
+def shield(package, type):
+	if type == "title":
+		url = "https://img.shields.io/badge/ContentDB-{}-{}" \
+			.format(urlescape(package.title), urlescape("#375a7f"))
+	elif type == "downloads":
+		#api_url = abs_url_for("api.package", author=package.author.username, name=package.name)
+		api_url = "https://content.minetest.net" + url_for("api.package", author=package.author.username, name=package.name)
+		url = "https://img.shields.io/badge/dynamic/json?color={}&label=ContentDB&query=downloads&suffix=+downloads&url={}" \
+			.format(urlescape("#375a7f"), urlescape(api_url))
+	else:
+		abort(404)
+
+	return redirect(url)
 
 
 @bp.route("/packages/<author>/<name>/download/")
@@ -167,30 +211,38 @@ def download(package):
 		return redirect(release.getDownloadURL(), code=302)
 
 
+def makeLabel(obj):
+	if obj.description:
+		return "{}: {}".format(obj.title, obj.description)
+	else:
+		return obj.title
+
 class PackageForm(FlaskForm):
-	name          = StringField("Name (Technical)", [InputRequired(), Length(1, 20), Regexp("^[a-z0-9_]+$", 0, "Lower case letters (a-z), digits (0-9), and underscores (_) only")])
-	title         = StringField("Title (Human-readable)", [InputRequired(), Length(3, 50)])
-	short_desc     = StringField("Short Description (Plaintext)", [InputRequired(), Length(1,200)])
-	desc          = TextAreaField("Long Description (Markdown)", [Optional(), Length(0,10000)])
-	type          = SelectField("Type", [InputRequired()], choices=PackageType.choices(), coerce=PackageType.coerce, default=PackageType.MOD)
-	license       = QuerySelectField("License", [DataRequired()], allow_blank=True, query_factory=lambda: License.query.order_by(db.asc(License.name)), get_pk=lambda a: a.id, get_label=lambda a: a.name)
-	media_license = QuerySelectField("Media License", [DataRequired()], allow_blank=True, query_factory=lambda: License.query.order_by(db.asc(License.name)), get_pk=lambda a: a.id, get_label=lambda a: a.name)
-	provides_str  = StringField("Provides (mods included in package)", [Optional()])
-	tags          = QuerySelectMultipleField('Tags', query_factory=lambda: Tag.query.order_by(db.asc(Tag.name)), get_pk=lambda a: a.id, get_label=lambda a: a.title)
-	harddep_str   = StringField("Hard Dependencies", [Optional()])
-	softdep_str   = StringField("Soft Dependencies", [Optional()])
-	repo          = StringField("VCS Repository URL", [Optional(), URL()], filters = [lambda x: x or None])
-	website       = StringField("Website URL", [Optional(), URL()], filters = [lambda x: x or None])
-	issueTracker  = StringField("Issue Tracker URL", [Optional(), URL()], filters = [lambda x: x or None])
-	forums	      = IntegerField("Forum Topic ID", [Optional(), NumberRange(0,999999)])
-	submit	      = SubmitField("Save")
+	type             = SelectField("Type", [InputRequired()], choices=PackageType.choices(), coerce=PackageType.coerce, default=PackageType.MOD)
+	title            = StringField("Title (Human-readable)", [InputRequired(), Length(3, 100)])
+	name             = StringField("Name (Technical)", [InputRequired(), Length(1, 100), Regexp("^[a-z0-9_]+$", 0, "Lower case letters (a-z), digits (0-9), and underscores (_) only")])
+	short_desc       = StringField("Short Description (Plaintext)", [InputRequired(), Length(1,200)])
+
+	tags             = QuerySelectMultipleField('Tags', query_factory=lambda: Tag.query.order_by(db.asc(Tag.name)), get_pk=lambda a: a.id, get_label=makeLabel)
+	content_warnings = QuerySelectMultipleField('Content Warnings', query_factory=lambda: ContentWarning.query.order_by(db.asc(ContentWarning.name)), get_pk=lambda a: a.id, get_label=makeLabel)
+	license          = QuerySelectField("License", [DataRequired()], allow_blank=True, query_factory=lambda: License.query.order_by(db.asc(License.name)), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	media_license    = QuerySelectField("Media License", [DataRequired()], allow_blank=True, query_factory=lambda: License.query.order_by(db.asc(License.name)), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+
+	desc             = TextAreaField("Long Description (Markdown)", [Optional(), Length(0,10000)])
+
+	repo             = StringField("VCS Repository URL", [Optional(), URL()], filters = [lambda x: x or None])
+	website          = StringField("Website URL", [Optional(), URL()], filters = [lambda x: x or None])
+	issueTracker     = StringField("Issue Tracker URL", [Optional(), URL()], filters = [lambda x: x or None])
+	forums           = IntegerField("Forum Topic ID", [Optional(), NumberRange(0,999999)])
+
+	submit           = SubmitField("Save")
+
 
 @bp.route("/packages/new/", methods=["GET", "POST"])
 @bp.route("/packages/<author>/<name>/edit/", methods=["GET", "POST"])
 @login_required
 def create_edit(author=None, name=None):
 	package = None
-	form = None
 	if author is None:
 		form = PackageForm(formdata=request.form)
 		author = request.args.get("author")
@@ -208,6 +260,8 @@ def create_edit(author=None, name=None):
 
 	else:
 		package = getPackageByInfo(author, name)
+		if package is None:
+			abort(404)
 		if not package.checkPerm(current_user, Permission.EDIT_PACKAGE):
 			return redirect(package.getDetailsURL())
 
@@ -225,16 +279,20 @@ def create_edit(author=None, name=None):
 			form.license.data = None
 			form.media_license.data = None
 		else:
-			form.harddep_str.data  = ",".join([str(x) for x in package.getSortedHardDependencies() ])
-			form.softdep_str.data  = ",".join([str(x) for x in package.getSortedOptionalDependencies() ])
-			form.provides_str.data = MetaPackage.ListToSpec(package.provides)
+			# form.harddep_str.data  = ",".join([str(x) for x in package.getSortedHardDependencies() ])
+			# form.softdep_str.data  = ",".join([str(x) for x in package.getSortedOptionalDependencies() ])
+			form.tags.data         = list(package.tags)
+			form.content_warnings.data = list(package.content_warnings)
 
-	if request.method == "POST" and form.validate():
+	if request.method == "POST" and form.type.data == PackageType.TXP:
+		form.license.data = form.media_license.data
+
+	if form.validate_on_submit():
 		wasNew = False
 		if not package:
 			package = Package.query.filter_by(name=form["name"].data, author_id=author.id).first()
 			if package is not None:
-				if package.soft_deleted:
+				if package.state == PackageState.READY_FOR_REVIEW:
 					Package.query.filter_by(name=form["name"].data, author_id=author.id).delete()
 				else:
 					flash("Package already exists!", "danger")
@@ -242,89 +300,87 @@ def create_edit(author=None, name=None):
 
 			package = Package()
 			package.author = author
+			package.maintainers.append(author)
 			wasNew = True
 
-		elif package.approved and package.name != form.name.data and \
-				not package.checkPerm(current_user, Permission.CHANGE_NAME):
-			flash("Unable to change package name", "danger")
-			return redirect(url_for("packages.create_edit", author=author, name=name))
+		try:
+			do_edit_package(current_user, package, wasNew, {
+				"type": form.type.data,
+				"title": form.title.data,
+				"name": form.name.data,
+				"short_desc": form.short_desc.data,
+				"tags": form.tags.raw_data,
+				"content_warnings": form.content_warnings.raw_data,
+				"license": form.license.data,
+				"media_license": form.media_license.data,
+				"desc": form.desc.data,
+				"repo": form.repo.data,
+				"website": form.website.data,
+				"issueTracker": form.issueTracker.data,
+				"forums": form.forums.data,
+			})
 
-		else:
-			triggerNotif(package.author, current_user,
-					"{} edited".format(package.title), package.getDetailsURL())
+			if wasNew and package.repo is not None:
+				importRepoScreenshot.delay(package.id)
 
-		form.populate_obj(package) # copy to row
+			next_url = package.getDetailsURL()
+			if wasNew and ("WTFPL" in package.license.name or "WTFPL" in package.media_license.name):
+				next_url = url_for("flatpage", path="help/wtfpl", r=next_url)
+			elif wasNew:
+				next_url = package.getSetupReleasesURL()
 
-		if package.type== PackageType.TXP:
-			package.license = package.media_license
+			return redirect(next_url)
+		except LogicError as e:
+			flash(e.message, "danger")
 
-		mpackage_cache = {}
-		package.provides.clear()
-		mpackages = MetaPackage.SpecToList(form.provides_str.data, mpackage_cache)
-		for m in mpackages:
-			package.provides.append(m)
-
-		Dependency.query.filter_by(depender=package).delete()
-		deps = Dependency.SpecToList(package, form.harddep_str.data, mpackage_cache)
-		for dep in deps:
-			dep.optional = False
-			db.session.add(dep)
-
-		deps = Dependency.SpecToList(package, form.softdep_str.data, mpackage_cache)
-		for dep in deps:
-			dep.optional = True
-			db.session.add(dep)
-
-		if wasNew and package.type == PackageType.MOD and not package.name in mpackage_cache:
-			m = MetaPackage.GetOrCreate(package.name, mpackage_cache)
-			package.provides.append(m)
-
-		package.tags.clear()
-		for tag in form.tags.raw_data:
-			package.tags.append(Tag.query.get(tag))
-
-		db.session.commit() # save
-
-		next_url = package.getDetailsURL()
-		if wasNew and package.repo is not None:
-			task = importRepoScreenshot.delay(package.id)
-			next_url = url_for("tasks.check", id=task.id, r=next_url)
-
-		if wasNew and ("WTFPL" in package.license.name or "WTFPL" in package.media_license.name):
-			next_url = url_for("flatpage", path="help/wtfpl", r=next_url)
-
-		return redirect(next_url)
-
-	package_query = Package.query.filter_by(approved=True, soft_deleted=False)
+	package_query = Package.query.filter_by(state=PackageState.APPROVED)
 	if package is not None:
 		package_query = package_query.filter(Package.id != package.id)
 
 	enableWizard = name is None and request.method != "POST"
-	return render_template("packages/create_edit.html", package=package, \
-			form=form, author=author, enable_wizard=enableWizard, \
-			packages=package_query.all(), \
+	return render_template("packages/create_edit.html", package=package,
+			form=form, author=author, enable_wizard=enableWizard,
+			packages=package_query.all(),
 			mpackages=MetaPackage.query.order_by(db.asc(MetaPackage.name)).all())
 
-@bp.route("/packages/<author>/<name>/approve/", methods=["POST"])
+
+@bp.route("/packages/<author>/<name>/state/", methods=["POST"])
 @login_required
 @is_package_page
-def approve(package):
-	if not package.checkPerm(current_user, Permission.APPROVE_NEW):
-		flash("You don't have permission to do that.", "danger")
+def move_to_state(package):
+	state = PackageState.get(request.args.get("state"))
+	if state is None:
+		abort(400)
 
-	elif package.approved:
-		flash("Package has already been approved", "danger")
+	if not package.canMoveToState(current_user, state):
+		flash("You don't have permission to do that", "danger")
+		return redirect(package.getDetailsURL())
 
-	else:
-		package.approved = True
+	package.state = state
+	msg = "Marked {} as {}".format(package.title, state.value)
+
+	if state == PackageState.APPROVED:
+		if not package.approved_at:
+			package.approved_at = datetime.datetime.now()
 
 		screenshots = PackageScreenshot.query.filter_by(package=package, approved=False).all()
 		for s in screenshots:
 			s.approved = True
 
-		triggerNotif(package.author, current_user,
-				"{} approved".format(package.title), package.getDetailsURL())
-		db.session.commit()
+		msg = "Approved {}".format(package.title)
+
+	addNotification(package.maintainers, current_user, NotificationType.PACKAGE_APPROVAL, msg, package.getDetailsURL(), package)
+	severity = AuditSeverity.NORMAL if current_user in package.maintainers else AuditSeverity.EDITOR
+	addAuditLog(severity, current_user, msg, package.getDetailsURL(), package)
+
+	db.session.commit()
+
+	if package.state == PackageState.CHANGES_NEEDED:
+		flash("Please comment what changes are needed in the review thread", "warning")
+		if package.review_thread:
+			return redirect(package.review_thread.getViewURL())
+		else:
+			return redirect(url_for('threads.new', pid=package.id, title='Package approval comments'))
 
 	return redirect(package.getDetailsURL())
 
@@ -341,11 +397,12 @@ def remove(package):
 			flash("You don't have permission to do that.", "danger")
 			return redirect(package.getDetailsURL())
 
-		package.soft_deleted = True
+		package.state = PackageState.DELETED
 
 		url = url_for("users.profile", username=package.author.username)
-		triggerNotif(package.author, current_user,
-				"{} deleted".format(package.title), url)
+		msg = "Deleted {}".format(package.title)
+		addNotification(package.maintainers, current_user, NotificationType.PACKAGE_EDIT, msg, url, package)
+		addAuditLog(AuditSeverity.EDITOR, current_user, msg, url)
 		db.session.commit()
 
 		flash("Deleted package", "success")
@@ -356,10 +413,12 @@ def remove(package):
 			flash("You don't have permission to do that.", "danger")
 			return redirect(package.getDetailsURL())
 
-		package.approved = False
+		package.state = PackageState.WIP
+
+		msg = "Unapproved {}".format(package.title)
+		addNotification(package.maintainers, current_user, NotificationType.PACKAGE_APPROVAL, msg, package.getDetailsURL(), package)
+		addAuditLog(AuditSeverity.EDITOR, current_user, msg, package.getDetailsURL(), package)
 
-		triggerNotif(package.author, current_user,
-				"{} deleted".format(package.title), package.getDetailsURL())
 		db.session.commit()
 
 		flash("Unapproved package", "success")
@@ -367,3 +426,104 @@ def remove(package):
 		return redirect(package.getDetailsURL())
 	else:
 		abort(400)
+
+
+
+class PackageMaintainersForm(FlaskForm):
+	maintainers_str  = StringField("Maintainers (Comma-separated)", [Optional()])
+	submit	      = SubmitField("Save")
+
+
+@bp.route("/packages/<author>/<name>/edit-maintainers/", methods=["GET", "POST"])
+@login_required
+@is_package_page
+def edit_maintainers(package):
+	if not package.checkPerm(current_user, Permission.EDIT_MAINTAINERS):
+		flash("You do not have permission to edit maintainers", "danger")
+		return redirect(package.getDetailsURL())
+
+	form = PackageMaintainersForm(formdata=request.form)
+	if request.method == "GET":
+		form.maintainers_str.data = ", ".join([ x.username for x in package.maintainers if x != package.author ])
+
+	if form.validate_on_submit():
+		usernames = [x.strip().lower() for x in form.maintainers_str.data.split(",")]
+		users = User.query.filter(func.lower(User.username).in_(usernames)).all()
+
+		for user in users:
+			if not user in package.maintainers:
+				addNotification(user, current_user, NotificationType.MAINTAINER,
+						"Added you as a maintainer of {}".format(package.title), package.getDetailsURL(), package)
+
+		for user in package.maintainers:
+			if user != package.author and not user in users:
+				addNotification(user, current_user, NotificationType.MAINTAINER,
+						"Removed you as a maintainer of {}".format(package.title), package.getDetailsURL(), package)
+
+		package.maintainers.clear()
+		package.maintainers.extend(users)
+		if package.author not in package.maintainers:
+			package.maintainers.append(package.author)
+
+		msg = "Edited {} maintainers".format(package.title)
+		addNotification(package.author, current_user, NotificationType.MAINTAINER, msg, package.getDetailsURL(), package)
+		severity = AuditSeverity.NORMAL if current_user == package.author else AuditSeverity.MODERATION
+		addAuditLog(severity, current_user, msg, package.getDetailsURL(), package)
+
+		db.session.commit()
+
+		return redirect(package.getDetailsURL())
+
+	users = User.query.filter(User.rank >= UserRank.NEW_MEMBER).order_by(db.asc(User.username)).all()
+
+	return render_template("packages/edit_maintainers.html",
+			package=package, form=form, users=users)
+
+
+@bp.route("/packages/<author>/<name>/remove-self-maintainer/", methods=["POST"])
+@login_required
+@is_package_page
+def remove_self_maintainers(package):
+	if not current_user in package.maintainers:
+		flash("You are not a maintainer", "danger")
+
+	elif current_user == package.author:
+		flash("Package owners cannot remove themselves as maintainers", "danger")
+
+	else:
+		package.maintainers.remove(current_user)
+
+		addNotification(package.author, current_user, NotificationType.MAINTAINER,
+				"Removed themself as a maintainer of {}".format(package.title), package.getDetailsURL(), package)
+
+		db.session.commit()
+
+	return redirect(package.getDetailsURL())
+
+
+@bp.route("/packages/<author>/<name>/import-meta/", methods=["POST"])
+@login_required
+@is_package_page
+def update_from_release(package):
+	if not package.checkPerm(current_user, Permission.REIMPORT_META):
+		flash("You don't have permission to reimport meta", "danger")
+		return redirect(package.getDetailsURL())
+
+	release = package.releases.first()
+	if not release:
+		flash("Release needed", "danger")
+		return redirect(package.getDetailsURL())
+
+	msg = "Updated meta from latest release"
+	addNotification(package.maintainers, current_user, NotificationType.PACKAGE_EDIT,
+			msg, package.getDetailsURL(), package)
+	severity = AuditSeverity.NORMAL if current_user in package.maintainers else AuditSeverity.EDITOR
+	addAuditLog(severity, current_user, msg, package.getDetailsURL(), package)
+
+	db.session.commit()
+
+	task_id = uuid()
+	zippath = release.url.replace("/uploads/", app.config["UPLOAD_DIR"])
+	checkZipRelease.apply_async((release.id, zippath), task_id=task_id)
+
+	return redirect(url_for("tasks.check", id=task_id, r=package.getEditURL()))

app/blueprints/packages/releases.py

@@ -1,35 +1,32 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
 from flask import *
-from flask_user import *
-
-from . import bp
-
-from app.rediscache import has_key, set_key, make_download_key
-from app.models import *
-from app.tasks.importtasks import makeVCSRelease, checkZipRelease
-from app.utils import *
-
-from celery import uuid
+from flask_login import login_required
 from flask_wtf import FlaskForm
 from wtforms import *
-from wtforms.validators import *
 from wtforms.ext.sqlalchemy.fields import QuerySelectField
+from wtforms.validators import *
+
+from app.logic.releases import do_create_vcs_release, LogicError, do_create_zip_release
+from app.rediscache import has_key, set_key, make_download_key
+from app.tasks.importtasks import check_update_config
+from app.utils import *
+from . import bp
 
 
 def get_mt_releases(is_max):
@@ -45,7 +42,7 @@ def get_mt_releases(is_max):
 class CreatePackageReleaseForm(FlaskForm):
 	title	   = StringField("Title", [InputRequired(), Length(1, 30)])
 	uploadOpt  = RadioField ("Method", choices=[("upload", "File Upload")], default="upload")
-	vcsLabel   = StringField("VCS Commit Hash, Branch, or Tag", default="master")
+	vcsLabel   = StringField("Git reference (ie: commit hash, branch, or tag)", default=None)
 	fileUpload = FileField("File Upload")
 	min_rel    = QuerySelectField("Minimum Minetest Version", [InputRequired()],
 			query_factory=lambda: get_mt_releases(False), get_pk=lambda a: a.id, get_label=lambda a: a.name)
@@ -55,7 +52,7 @@ class CreatePackageReleaseForm(FlaskForm):
 
 class EditPackageReleaseForm(FlaskForm):
 	title    = StringField("Title", [InputRequired(), Length(1, 30)])
-	url      = StringField("URL", [URL])
+	url      = StringField("URL", [Optional()])
 	task_id  = StringField("Task ID", filters = [lambda x: x or None])
 	approved = BooleanField("Is Approved")
 	min_rel  = QuerySelectField("Minimum Minetest Version", [InputRequired()],
@@ -74,52 +71,29 @@ def create_release(package):
 	# Initial form class from post data and default data
 	form = CreatePackageReleaseForm()
 	if package.repo is not None:
-		form["uploadOpt"].choices = [("vcs", "From Git Commit or Branch"), ("upload", "File Upload")]
-		if request.method != "POST":
+		form["uploadOpt"].choices = [("vcs", "Import from Git"), ("upload", "Upload .zip file")]
+		if request.method == "GET":
 			form["uploadOpt"].data = "vcs"
+			form.vcsLabel.data = request.args.get("ref")
 
-	if request.method == "POST" and form.validate():
-		if form["uploadOpt"].data == "vcs":
-			rel = PackageRelease()
-			rel.package = package
-			rel.title   = form["title"].data
-			rel.url     = ""
-			rel.task_id = uuid()
-			rel.min_rel = form["min_rel"].data.getActual()
-			rel.max_rel = form["max_rel"].data.getActual()
-			db.session.add(rel)
-			db.session.commit()
-
-			makeVCSRelease.apply_async((rel.id, form["vcsLabel"].data), task_id=rel.task_id)
-
-			msg = "{}: Release {} created".format(package.title, rel.title)
-			triggerNotif(package.author, current_user, msg, rel.getEditURL())
-			db.session.commit()
+	if request.method == "GET":
+		form.title.data = request.args.get("title")
 
+	if form.validate_on_submit():
+		try:
+			if form["uploadOpt"].data == "vcs":
+				rel = do_create_vcs_release(current_user, package, form.title.data,
+						form.vcsLabel.data, form.min_rel.data.getActual(), form.max_rel.data.getActual())
+			else:
+				rel = do_create_zip_release(current_user, package, form.title.data,
+						form.fileUpload.data, form.min_rel.data.getActual(), form.max_rel.data.getActual())
 			return redirect(url_for("tasks.check", id=rel.task_id, r=rel.getEditURL()))
-		else:
-			uploadedUrl, uploadedPath = doFileUpload(form.fileUpload.data, "zip", "a zip file")
-			if uploadedUrl is not None:
-				rel = PackageRelease()
-				rel.package = package
-				rel.title = form["title"].data
-				rel.url = uploadedUrl
-				rel.task_id = uuid()
-				rel.min_rel = form["min_rel"].data.getActual()
-				rel.max_rel = form["max_rel"].data.getActual()
-				db.session.add(rel)
-				db.session.commit()
-
-				checkZipRelease.apply_async((rel.id, uploadedPath), task_id=rel.task_id)
-
-				msg = "{}: Release {} created".format(package.title, rel.title)
-				triggerNotif(package.author, current_user, msg, rel.getEditURL())
-				db.session.commit()
-
-				return redirect(url_for("tasks.check", id=rel.task_id, r=rel.getEditURL()))
+		except LogicError as e:
+			flash(e.message, "danger")
 
 	return render_template("packages/release_new.html", package=package, form=form)
 
+
 @bp.route("/packages/<author>/<name>/releases/<id>/download/")
 @is_package_page
 def download_release(package, id):
@@ -128,20 +102,20 @@ def download_release(package, id):
 		abort(404)
 
 	ip = request.headers.get("X-Forwarded-For") or request.remote_addr
-	if ip is not None:
+	if ip is not None and not is_user_bot():
 		key = make_download_key(ip, release.package)
 		if not has_key(key):
 			set_key(key, "true")
 
 			bonus = 1
-			if not package.getIsFOSS():
-				bonus *= 0.1
 
 			PackageRelease.query.filter_by(id=release.id).update({
 					"downloads": PackageRelease.downloads + 1
 				})
 
 			Package.query.filter_by(id=package.id).update({
+					"downloads": Package.downloads + 1,
+					"score_downloads": Package.score_downloads + bonus,
 					"score": Package.score + bonus
 				})
 
@@ -149,16 +123,15 @@ def download_release(package, id):
 
 	return redirect(release.url, code=300)
 
+
 @bp.route("/packages/<author>/<name>/releases/<id>/", methods=["GET", "POST"])
 @login_required
 @is_package_page
 def edit_release(package, id):
-	release = PackageRelease.query.get(id)
+	release : PackageRelease = PackageRelease.query.get(id)
 	if release is None or release.package != package:
 		abort(404)
 
-	clearNotifications(release.getEditURL())
-
 	canEdit	= package.checkPerm(current_user, Permission.MAKE_RELEASE)
 	canApprove = package.checkPerm(current_user, Permission.APPROVE_RELEASE)
 	if not (canEdit or canApprove):
@@ -167,12 +140,11 @@ def edit_release(package, id):
 	# Initial form class from post data and default data
 	form = EditPackageReleaseForm(formdata=request.form, obj=release)
 
-	# HACK: fix bug in wtforms
 	if request.method == "GET":
+		# HACK: fix bug in wtforms
 		form.approved.data = release.approved
 
-	if request.method == "POST" and form.validate():
-		wasApproved = release.approved
+	if form.validate_on_submit():
 		if canEdit:
 			release.title = form["title"].data
 			release.min_rel = form["min_rel"].data.getActual()
@@ -184,10 +156,10 @@ def edit_release(package, id):
 			if release.task_id is not None:
 				release.task_id = None
 
-		if canApprove:
-			release.approved = form["approved"].data
-		else:
-			release.approved = wasApproved
+		if form.approved.data:
+			release.approve(current_user)
+		elif canApprove:
+			release.approved = False
 
 		db.session.commit()
 		return redirect(package.getDetailsURL())
@@ -219,7 +191,7 @@ def bulk_change_release(package):
 
 	if request.method == "GET":
 		form.only_change_none.data = True
-	elif request.method == "POST" and form.validate():
+	elif form.validate_on_submit():
 		only_change_none = form.only_change_none.data
 
 		for release in package.releases.all():
@@ -250,3 +222,119 @@ def delete_release(package, id):
 	db.session.commit()
 
 	return redirect(package.getDetailsURL())
+
+
+class PackageUpdateConfigFrom(FlaskForm):
+	trigger = RadioField("Trigger", [InputRequired()], choices=PackageUpdateTrigger.choices(), coerce=PackageUpdateTrigger.coerce,
+			default=PackageUpdateTrigger.TAG)
+	ref     = StringField("Branch name", [Optional()], default=None)
+	action  = RadioField("Action", [InputRequired()], choices=[("notification", "Send notification and mark as outdated"), ("make_release", "Create release")], default="make_release")
+	submit  = SubmitField("Save Settings")
+	disable = SubmitField("Disable Automation")
+
+
+def set_update_config(package, form):
+	if package.update_config is None:
+		package.update_config = PackageUpdateConfig()
+		db.session.add(package.update_config)
+
+	form.populate_obj(package.update_config)
+	package.update_config.ref = nonEmptyOrNone(form.ref.data)
+	package.update_config.make_release = form.action.data == "make_release"
+
+	if package.update_config.trigger == PackageUpdateTrigger.COMMIT:
+		if package.update_config.last_commit is None:
+			last_release = package.releases.first()
+			if last_release and last_release.commit_hash:
+				package.update_config.last_commit = last_release.commit_hash
+	elif package.update_config.trigger == PackageUpdateTrigger.TAG:
+		# Only create releases for tags created after this
+		package.update_config.last_commit = None
+		package.update_config.last_tag = None
+
+	package.update_config.outdated_at = None
+	package.update_config.auto_created = False
+
+	db.session.commit()
+
+	if package.update_config.last_commit is None:
+		check_update_config.delay(package.id)
+
+
+@bp.route("/packages/<author>/<name>/update-config/", methods=["GET", "POST"])
+@login_required
+@is_package_page
+def update_config(package):
+	if not package.checkPerm(current_user, Permission.MAKE_RELEASE):
+		abort(403)
+
+	if not package.repo:
+		flash("Please add a Git repository URL in order to set up automatic releases", "danger")
+		return redirect(package.getEditURL())
+
+	form = PackageUpdateConfigFrom(obj=package.update_config)
+	if request.method == "GET":
+		if package.update_config:
+			form.action.data = "make_release" if package.update_config.make_release else "notification"
+		elif request.args.get("action") == "notification":
+			form.trigger.data = PackageUpdateTrigger.COMMIT
+			form.action.data = "notification"
+
+	if form.validate_on_submit():
+		if form.disable.data:
+			flash("Deleted update configuration", "success")
+			if package.update_config:
+				db.session.delete(package.update_config)
+			db.session.commit()
+		else:
+			set_update_config(package, form)
+
+		if not form.disable.data and package.releases.count() == 0:
+			flash("Now, please create an initial release", "success")
+			return redirect(package.getCreateReleaseURL())
+
+		return redirect(package.getDetailsURL())
+
+	return render_template("packages/update_config.html", package=package, form=form)
+
+
+@bp.route("/packages/<author>/<name>/setup-releases/")
+@login_required
+@is_package_page
+def setup_releases(package):
+	if not package.checkPerm(current_user, Permission.MAKE_RELEASE):
+		abort(403)
+
+	if package.update_config:
+		return redirect(package.getUpdateConfigURL())
+
+	return render_template("packages/release_wizard.html", package=package)
+
+
+@bp.route("/user/update-configs/")
+@bp.route("/users/<username>/update-configs/", methods=["GET", "POST"])
+@login_required
+def bulk_update_config(username=None):
+	if username is None:
+		return redirect(url_for("packages.bulk_update_config", username=current_user.username))
+
+	user: User = User.query.filter_by(username=username).first()
+	if not user:
+		abort(404)
+
+	if current_user != user and not current_user.rank.atLeast(UserRank.EDITOR):
+		abort(403)
+
+	form = PackageUpdateConfigFrom()
+	if form.validate_on_submit():
+		for package in user.packages.filter(Package.state != PackageState.DELETED, Package.repo.isnot(None)).all():
+			set_update_config(package, form)
+
+		return redirect(url_for("packages.bulk_update_config", username=username))
+
+	confs = user.packages \
+		.filter(Package.state != PackageState.DELETED,
+			Package.update_config.has()) \
+		.order_by(db.asc(Package.title)).all()
+
+	return render_template("packages/bulk_update_conf.html", user=user, confs=confs, form=form)

app/blueprints/packages/reviews.py

@@ -0,0 +1,143 @@
+# ContentDB
+# Copyright (C) 2020  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+from . import bp
+
+from flask import *
+from flask_login import current_user, login_required
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+from app.models import db, PackageReview, Thread, ThreadReply, NotificationType
+from app.utils import is_package_page, addNotification, get_int_or_abort
+
+
+@bp.route("/reviews/")
+def list_reviews():
+	page = get_int_or_abort(request.args.get("page"), 1)
+	num = min(40, get_int_or_abort(request.args.get("n"), 100))
+
+	pagination = PackageReview.query.order_by(db.desc(PackageReview.created_at)).paginate(page, num, True)
+	return render_template("packages/reviews_list.html", pagination=pagination, reviews=pagination.items)
+
+
+class ReviewForm(FlaskForm):
+	title	= StringField("Title", [InputRequired(), Length(3,100)])
+	comment = TextAreaField("Comment", [InputRequired(), Length(10, 2000)])
+	recommends = RadioField("Private", [InputRequired()], choices=[("yes", "Yes"), ("no", "No")])
+	submit  = SubmitField("Save")
+
+@bp.route("/packages/<author>/<name>/review/", methods=["GET", "POST"])
+@login_required
+@is_package_page
+def review(package):
+	if current_user in package.maintainers:
+		flash("You can't review your own package!", "danger")
+		return redirect(package.getDetailsURL())
+
+	review = PackageReview.query.filter_by(package=package, author=current_user).first()
+
+	form = ReviewForm(formdata=request.form, obj=review)
+
+	# Set default values
+	if request.method == "GET" and review:
+		form.title.data = review.thread.title
+		form.recommends.data = "yes" if review.recommends else "no"
+		form.comment.data = review.thread.replies[0].comment
+
+	# Validate and submit
+	elif form.validate_on_submit():
+		was_new = False
+		if not review:
+			was_new = True
+			review = PackageReview()
+			review.package = package
+			review.author  = current_user
+			db.session.add(review)
+
+		review.recommends = form.recommends.data == "yes"
+
+		thread = review.thread
+		if not thread:
+			thread = Thread()
+			thread.author  = current_user
+			thread.private = False
+			thread.package = package
+			thread.review = review
+			db.session.add(thread)
+
+			thread.watchers.append(current_user)
+
+			reply = ThreadReply()
+			reply.thread  = thread
+			reply.author  = current_user
+			reply.comment = form.comment.data
+			db.session.add(reply)
+
+			thread.replies.append(reply)
+		else:
+			reply = thread.replies[0]
+			reply.comment = form.comment.data
+
+		thread.title   = form.title.data
+
+		db.session.commit()
+
+		package.recalcScore()
+
+		if was_new:
+			notif_msg = "New review '{}'".format(form.title.data)
+			type = NotificationType.NEW_REVIEW
+		else:
+			notif_msg = "Updated review '{}'".format(form.title.data)
+			type = NotificationType.OTHER
+
+		addNotification(package.maintainers, current_user, type, notif_msg,
+				url_for("threads.view", id=thread.id), package)
+
+		db.session.commit()
+
+		return redirect(package.getDetailsURL())
+
+	return render_template("packages/review_create_edit.html",
+			form=form, package=package, review=review)
+
+
+@bp.route("/packages/<author>/<name>/review/delete/", methods=["POST"])
+@login_required
+@is_package_page
+def delete_review(package):
+	review = PackageReview.query.filter_by(package=package, author=current_user).first()
+	if review is None or review.package != package:
+		abort(404)
+
+	thread = review.thread
+
+	reply = ThreadReply()
+	reply.thread  = thread
+	reply.author  = current_user
+	reply.comment = "_converted review into a thread_"
+	db.session.add(reply)
+
+	thread.review = None
+
+	notif_msg = "Deleted review '{}', comments were kept as a thread".format(thread.title)
+	addNotification(package.maintainers, current_user, NotificationType.OTHER, notif_msg, url_for("threads.view", id=thread.id), package)
+
+	db.session.delete(review)
+	db.session.commit()
+
+	return redirect(thread.getViewURL())

app/blueprints/packages/screenshots.py

@@ -1,73 +1,98 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
 from flask import *
-from flask_user import *
-
-from . import bp
-
-from app.models import *
-from app.utils import *
-
 from flask_wtf import FlaskForm
+from flask_login import login_required
 from wtforms import *
+from wtforms.ext.sqlalchemy.fields import QuerySelectField
 from wtforms.validators import *
 
+from app.utils import *
+from . import bp
+from app.logic.LogicError import LogicError
+from app.logic.screenshots import do_create_screenshot, do_order_screenshots
+
 
 class CreateScreenshotForm(FlaskForm):
-	title	   = StringField("Title/Caption", [Optional()])
+	title	   = StringField("Title/Caption", [Optional(), Length(-1, 100)])
 	fileUpload = FileField("File Upload", [InputRequired()])
 	submit	   = SubmitField("Save")
 
 
 class EditScreenshotForm(FlaskForm):
-	title	 = StringField("Title/Caption", [Optional()])
+	title	 = StringField("Title/Caption", [Optional(), Length(-1, 100)])
 	approved = BooleanField("Is Approved")
-	delete   = BooleanField("Delete")
 	submit   = SubmitField("Save")
 
+
+class EditPackageScreenshotsForm(FlaskForm):
+	cover_image      = QuerySelectField("Cover Image", [DataRequired()], allow_blank=True, get_pk=lambda a: a.id, get_label=lambda a: a.title)
+	submit	         = SubmitField("Save")
+
+
+@bp.route("/packages/<author>/<name>/screenshots/", methods=["GET", "POST"])
+@login_required
+@is_package_page
+def screenshots(package):
+	if not package.checkPerm(current_user, Permission.ADD_SCREENSHOTS):
+		return redirect(package.getDetailsURL())
+
+	if package.screenshots.count() == 0:
+		return redirect(package.getNewScreenshotURL())
+
+	form = EditPackageScreenshotsForm(obj=package)
+	form.cover_image.query = package.screenshots
+
+	if request.method == "POST":
+		order = request.form.get("order")
+		if order:
+			try:
+				do_order_screenshots(current_user, package, order.split(","))
+				return redirect(package.getDetailsURL())
+			except LogicError as e:
+				flash(e.message, "danger")
+
+		if form.validate_on_submit():
+			form.populate_obj(package)
+			db.session.commit()
+
+	return render_template("packages/screenshots.html", package=package, form=form)
+
+
 @bp.route("/packages/<author>/<name>/screenshots/new/", methods=["GET", "POST"])
 @login_required
 @is_package_page
-def create_screenshot(package, id=None):
+def create_screenshot(package):
 	if not package.checkPerm(current_user, Permission.ADD_SCREENSHOTS):
 		return redirect(package.getDetailsURL())
 
 	# Initial form class from post data and default data
 	form = CreateScreenshotForm()
-	if request.method == "POST" and form.validate():
-		uploadedUrl, uploadedPath = doFileUpload(form.fileUpload.data, "image",
-				"a PNG or JPG image file")
-		if uploadedUrl is not None:
-			ss = PackageScreenshot()
-			ss.package  = package
-			ss.title    = form["title"].data or "Untitled"
-			ss.url      = uploadedUrl
-			ss.approved = package.checkPerm(current_user, Permission.APPROVE_SCREENSHOT)
-			db.session.add(ss)
-
-			msg = "{}: Screenshot added {}" \
-					.format(package.title, ss.title)
-			triggerNotif(package.author, current_user, msg, package.getDetailsURL())
-			db.session.commit()
-			return redirect(package.getDetailsURL())
+	if form.validate_on_submit():
+		try:
+			do_create_screenshot(current_user, package, form.title.data, form.fileUpload.data)
+			return redirect(package.getEditScreenshotsURL())
+		except LogicError as e:
+			flash(e.message, "danger")
 
 	return render_template("packages/screenshot_new.html", package=package, form=form)
 
+
 @bp.route("/packages/<author>/<name>/screenshots/<id>/edit/", methods=["GET", "POST"])
 @login_required
 @is_package_page
@@ -79,28 +104,44 @@ def edit_screenshot(package, id):
 	canEdit	= package.checkPerm(current_user, Permission.ADD_SCREENSHOTS)
 	canApprove = package.checkPerm(current_user, Permission.APPROVE_SCREENSHOT)
 	if not (canEdit or canApprove):
-		return redirect(package.getDetailsURL())
-
-	clearNotifications(screenshot.getEditURL())
+		return redirect(package.getEditScreenshotsURL())
 
 	# Initial form class from post data and default data
-	form = EditScreenshotForm(formdata=request.form, obj=screenshot)
-	if request.method == "POST" and form.validate():
-		if canEdit and form["delete"].data:
-			PackageScreenshot.query.filter_by(id=id).delete()
+	form = EditScreenshotForm(obj=screenshot)
+	if form.validate_on_submit():
+		wasApproved = screenshot.approved
 
+		if canEdit:
+			screenshot.title = form["title"].data or "Untitled"
+
+		if canApprove:
+			screenshot.approved = form["approved"].data
 		else:
-			wasApproved = screenshot.approved
-
-			if canEdit:
-				screenshot.title = form["title"].data or "Untitled"
-
-			if canApprove:
-				screenshot.approved = form["approved"].data
-			else:
-				screenshot.approved = wasApproved
+			screenshot.approved = wasApproved
 
 		db.session.commit()
-		return redirect(package.getDetailsURL())
+		return redirect(package.getEditScreenshotsURL())
 
 	return render_template("packages/screenshot_edit.html", package=package, screenshot=screenshot, form=form)
+
+
+@bp.route("/packages/<author>/<name>/screenshots/<id>/delete/", methods=["POST"])
+@login_required
+@is_package_page
+def delete_screenshot(package, id):
+	screenshot = PackageScreenshot.query.get(id)
+	if screenshot is None or screenshot.package != package:
+		abort(404)
+
+	if not package.checkPerm(current_user, Permission.ADD_SCREENSHOTS):
+		flash("Permission denied", "danger")
+		return redirect(url_for("homepage.home"))
+
+	if package.cover_image == screenshot:
+		package.cover_image = None
+		db.session.merge(package)
+
+	db.session.delete(screenshot)
+	db.session.commit()
+
+	return redirect(package.getEditScreenshotsURL())

app/blueprints/tasks/__init__.py

@@ -1,28 +1,26 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
 from flask import *
-from flask_user import *
-import flask_menu as menu
+from flask_login import login_required
+
 from app import csrf
-from app.models import *
-from app.tasks import celery, TaskError
+from app.tasks import celery
 from app.tasks.importtasks import getMeta
-from app.utils import shouldReturnJson
 from app.utils import *
 
 bp = Blueprint("tasks", __name__)
@@ -45,7 +43,7 @@ def check(id):
 	traceback = result.traceback
 	result = result.result
 
-	info = None
+	None
 	if isinstance(result, Exception):
 		info = {
 				'id': id,

app/blueprints/threads/__init__.py

@@ -1,35 +1,32 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-
 from flask import *
 
 bp = Blueprint("threads", __name__)
 
-from flask_user import *
+from flask_login import current_user, login_required
+from app import menu
 from app.models import *
-from app.utils import triggerNotif, clearNotifications
-
-import datetime
-
+from app.utils import addNotification, isYes, addAuditLog
 from flask_wtf import FlaskForm
 from wtforms import *
 from wtforms.validators import *
 from app.utils import get_int_or_abort
 
+@menu.register_menu(bp, ".threads", "Threads", order=20)
 @bp.route("/threads/")
 def list_all():
 	query = Thread.query
@@ -41,7 +38,16 @@ def list_all():
 		pid = get_int_or_abort(pid)
 		query = query.filter_by(package_id=pid)
 
-	return render_template("threads/list.html", threads=query.all())
+	query = query.filter_by(review_id=None)
+
+	query = query.order_by(db.desc(Thread.created_at))
+
+	page = get_int_or_abort(request.args.get("page"), 1)
+	num = min(40, get_int_or_abort(request.args.get("n"), 100))
+
+	pagination = query.paginate(page, num, True)
+
+	return render_template("threads/list.html", pagination=pagination, threads=pagination.items)
 
 
 @bp.route("/threads/<int:id>/subscribe/", methods=["POST"])
@@ -58,7 +64,7 @@ def subscribe(id):
 		thread.watchers.append(current_user)
 		db.session.commit()
 
-	return redirect(url_for("threads.view", id=id))
+	return redirect(thread.getViewURL())
 
 
 @bp.route("/threads/<int:id>/unsubscribe/", methods=["POST"])
@@ -73,15 +79,138 @@ def unsubscribe(id):
 		thread.watchers.remove(current_user)
 		db.session.commit()
 	else:
-		flash("Not subscribed to thread", "success")
+		flash("Already not subscribed!", "success")
 
-	return redirect(url_for("threads.view", id=id))
+	return redirect(thread.getViewURL())
+
+
+@bp.route("/threads/<int:id>/set-lock/", methods=["POST"])
+@login_required
+def set_lock(id):
+	thread = Thread.query.get(id)
+	if thread is None or not thread.checkPerm(current_user, Permission.LOCK_THREAD):
+		abort(404)
+
+	thread.locked = isYes(request.args.get("lock"))
+	if thread.locked is None:
+		abort(400)
+
+	msg = None
+	if thread.locked:
+		msg = "Locked thread '{}'".format(thread.title)
+		flash("Locked thread", "success")
+	else:
+		msg = "Unlocked thread '{}'".format(thread.title)
+		flash("Unlocked thread", "success")
+
+	addNotification(thread.watchers, current_user, NotificationType.OTHER, msg, thread.getViewURL(), thread.package)
+	addAuditLog(AuditSeverity.MODERATION, current_user, NotificationType.OTHER, msg, thread.getViewURL(), thread.package)
+
+	db.session.commit()
+
+	return redirect(thread.getViewURL())
+
+
+@bp.route("/threads/<int:id>/delete/", methods=["GET", "POST"])
+@login_required
+def delete_thread(id):
+	thread = Thread.query.get(id)
+	if thread is None or not thread.checkPerm(current_user, Permission.DELETE_THREAD):
+		abort(404)
+
+	if request.method == "GET":
+		return render_template("threads/delete_thread.html", thread=thread)
+
+	summary = "\n\n".join([("<{}> {}".format(reply.author.display_name, reply.comment)) for reply in thread.replies])
+
+	msg = "Deleted thread {} by {}".format(thread.title, thread.author.display_name)
+
+	db.session.delete(thread)
+
+	addAuditLog(AuditSeverity.MODERATION, current_user, msg, None, thread.package, summary)
+
+	db.session.commit()
+
+	return redirect(url_for("homepage.home"))
+
+
+@bp.route("/threads/<int:id>/delete-reply/", methods=["GET", "POST"])
+@login_required
+def delete_reply(id):
+	thread = Thread.query.get(id)
+	if thread is None:
+		abort(404)
+
+	reply_id = request.args.get("reply")
+	if reply_id is None:
+		abort(404)
+
+	reply = ThreadReply.query.get(reply_id)
+	if reply is None or reply.thread != thread:
+		abort(404)
+
+	if thread.replies[0] == reply:
+		flash("Cannot delete thread opening post!", "danger")
+		return redirect(thread.getViewURL())
+
+	if not reply.checkPerm(current_user, Permission.DELETE_REPLY):
+		abort(403)
+
+	if request.method == "GET":
+		return render_template("threads/delete_reply.html", thread=thread, reply=reply)
+
+	msg = "Deleted reply by {}".format(reply.author.display_name)
+	addAuditLog(AuditSeverity.MODERATION, current_user, msg, thread.getViewURL(), thread.package, reply.comment)
+
+	db.session.delete(reply)
+	db.session.commit()
+
+	return redirect(thread.getViewURL())
+
+
+class CommentForm(FlaskForm):
+	comment = TextAreaField("Comment", [InputRequired(), Length(10, 2000)])
+	submit  = SubmitField("Comment")
+
+
+@bp.route("/threads/<int:id>/edit/", methods=["GET", "POST"])
+@login_required
+def edit_reply(id):
+	thread = Thread.query.get(id)
+	if thread is None:
+		abort(404)
+
+	reply_id = request.args.get("reply")
+	if reply_id is None:
+		abort(404)
+
+	reply = ThreadReply.query.get(reply_id)
+	if reply is None or reply.thread != thread:
+		abort(404)
+
+	if not reply.checkPerm(current_user, Permission.EDIT_REPLY):
+		abort(403)
+
+	form = CommentForm(formdata=request.form, obj=reply)
+	if form.validate_on_submit():
+		comment = form.comment.data
+
+		msg = "Edited reply by {}".format(reply.author.display_name)
+		severity = AuditSeverity.NORMAL if current_user == reply.author else AuditSeverity.MODERATION
+		addNotification(reply.author, current_user, NotificationType.OTHER, msg, thread.getViewURL(), thread.package)
+		addAuditLog(severity, current_user, msg, thread.getViewURL(), thread.package, reply.comment)
+
+		reply.comment = comment
+
+		db.session.commit()
+
+		return redirect(thread.getViewURL())
+
+	return render_template("threads/edit_reply.html", thread=thread, reply=reply, form=form)
 
 
 @bp.route("/threads/<int:id>/", methods=["GET", "POST"])
 def view(id):
-	clearNotifications(url_for("threads.view", id=id))
-
 	thread = Thread.query.get(id)
 	if thread is None or not thread.checkPerm(current_user, Permission.SEE_THREAD):
 		abort(404)
@@ -89,14 +218,15 @@ def view(id):
 	if current_user.is_authenticated and request.method == "POST":
 		comment = request.form["comment"]
 
+		if not thread.checkPerm(current_user, Permission.COMMENT_THREAD):
+			flash("You cannot comment on this thread", "danger")
+			return redirect(thread.getViewURL())
+
 		if not current_user.canCommentRL():
 			flash("Please wait before commenting again", "danger")
-			if package:
-				return redirect(package.getDetailsURL())
-			else:
-				return redirect(url_for("homepage.home"))
+			return redirect(thread.getViewURL())
 
-		if len(comment) <= 500 and len(comment) > 3:
+		if 2000 >= len(comment) > 3:
 			reply = ThreadReply()
 			reply.author = current_user
 			reply.comment = comment
@@ -106,33 +236,25 @@ def view(id):
 			if not current_user in thread.watchers:
 				thread.watchers.append(current_user)
 
-			msg = None
-			if thread.package is None:
-				msg = "New comment on '{}'".format(thread.title)
-			else:
-				msg = "New comment on '{}' on package {}".format(thread.title, thread.package.title)
-
-
-			for user in thread.watchers:
-				if user != current_user:
-					triggerNotif(user, current_user, msg, url_for("threads.view", id=thread.id))
-
+			msg = "New comment on '{}'".format(thread.title)
+			addNotification(thread.watchers, current_user, NotificationType.THREAD_REPLY, msg, thread.getViewURL(), thread.package)
 			db.session.commit()
 
-			return redirect(url_for("threads.view", id=id))
+			return redirect(thread.getViewURL())
 
 		else:
-			flash("Comment needs to be between 3 and 500 characters.")
+			flash("Comment needs to be between 3 and 2000 characters.")
 
 	return render_template("threads/view.html", thread=thread)
 
 
 class ThreadForm(FlaskForm):
 	title	= StringField("Title", [InputRequired(), Length(3,100)])
-	comment = TextAreaField("Comment", [InputRequired(), Length(10, 500)])
+	comment = TextAreaField("Comment", [InputRequired(), Length(10, 2000)])
 	private = BooleanField("Private")
 	submit  = SubmitField("Open Thread")
 
+
 @bp.route("/threads/new/", methods=["GET", "POST"])
 @login_required
 def new():
@@ -162,7 +284,7 @@ def new():
 	# Only allow creating one thread when not approved
 	elif is_review_thread and package.review_thread is not None:
 		flash("A review thread already exists!", "danger")
-		return redirect(url_for("threads.view", id=package.review_thread.id))
+		return redirect(package.review_thread.getViewURL())
 
 	elif not current_user.canOpenThreadRL():
 		flash("Please wait before opening another thread", "danger")
@@ -178,7 +300,7 @@ def new():
 		form.title.data   = request.args.get("title") or ""
 
 	# Validate and submit
-	elif request.method == "POST" and form.validate():
+	elif form.validate_on_submit():
 		thread = Thread()
 		thread.author  = current_user
 		thread.title   = form.title.data
@@ -203,19 +325,20 @@ def new():
 		if is_review_thread:
 			package.review_thread = thread
 
-		notif_msg = None
-		if package is not None:
-			notif_msg = "New thread '{}' on package {}".format(thread.title, package.title)
-			triggerNotif(package.author, current_user, notif_msg, url_for("threads.view", id=thread.id))
-		else:
-			notif_msg = "New thread '{}'".format(thread.title)
+			if package.state == PackageState.READY_FOR_REVIEW and current_user not in package.maintainers:
+				package.state = PackageState.CHANGES_NEEDED
 
-		for user in User.query.filter(User.rank >= UserRank.EDITOR).all():
-			triggerNotif(user, current_user, notif_msg, url_for("threads.view", id=thread.id))
+
+		notif_msg = "New thread '{}'".format(thread.title)
+		if package is not None:
+			addNotification(package.maintainers, current_user, NotificationType.NEW_THREAD, notif_msg, thread.getViewURL(), package)
+
+		editors = User.query.filter(User.rank >= UserRank.EDITOR).all()
+		addNotification(editors, current_user, NotificationType.EDITOR_MISC, notif_msg, thread.getViewURL(), package)
 
 		db.session.commit()
 
-		return redirect(url_for("threads.view", id=thread.id))
+		return redirect(thread.getViewURL())
 
 
 	return render_template("threads/new.html", form=form, allow_private_change=allow_change, package=package)

app/blueprints/thumbnails/__init__.py

@@ -1,21 +1,21 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-from flask import *
+from flask import abort, send_file, Blueprint, current_app
 
 bp = Blueprint("thumbnails", __name__)
 
@@ -26,12 +26,18 @@ ALLOWED_RESOLUTIONS=[(100,67), (270,180), (350,233)]
 
 def mkdir(path):
 	assert path != "" and path is not None
-	if not os.path.isdir(path):
-		os.mkdir(path)
+	try:
+		if not os.path.isdir(path):
+			os.mkdir(path)
+	except FileExistsError:
+		pass
 
 
 def resize_and_crop(img_path, modified_path, size):
-	img = Image.open(img_path)
+	try:
+		img = Image.open(img_path)
+	except FileNotFoundError:
+		abort(404)
 
 	# Get current and desired ratio for the images
 	img_ratio = img.size[0] / float(img.size[1])

app/blueprints/todo/__init__.py

@@ -1,38 +1,45 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
+from celery import uuid
 from flask import *
-from flask_user import *
-import flask_menu as menu
+from flask_login import current_user, login_required
+from sqlalchemy import or_
+
 from app.models import *
 from app.querybuilder import QueryBuilder
-from app.utils import get_int_or_abort
+from app.utils import get_int_or_abort, addNotification, addAuditLog, isYes
+from app.tasks.importtasks import makeVCSRelease
 
 bp = Blueprint("todo", __name__)
 
 @bp.route("/todo/", methods=["GET", "POST"])
 @login_required
-def view():
+def view_editor():
 	canApproveNew = Permission.APPROVE_NEW.check(current_user)
 	canApproveRel = Permission.APPROVE_RELEASE.check(current_user)
 	canApproveScn = Permission.APPROVE_SCREENSHOT.check(current_user)
 
 	packages = None
+	wip_packages = None
 	if canApproveNew:
-		packages = Package.query.filter_by(approved=False, soft_deleted=False).order_by(db.desc(Package.created_at)).all()
+		packages = Package.query.filter_by(state=PackageState.READY_FOR_REVIEW) \
+			.order_by(db.desc(Package.created_at)).all()
+		wip_packages = Package.query.filter(or_(Package.state==PackageState.WIP, Package.state==PackageState.CHANGES_NEEDED)) \
+			.order_by(db.desc(Package.created_at)).all()
 
 	releases = None
 	if canApproveRel:
@@ -52,22 +59,23 @@ def view():
 
 			PackageScreenshot.query.update({ "approved": True })
 			db.session.commit()
-			return redirect(url_for("todo.view"))
+			return redirect(url_for("todo.view_editor"))
 		else:
 			abort(400)
 
-	topic_query = ForumTopic.query \
-			.filter_by(discarded=False)
+	total_packages = Package.query.filter_by(state=PackageState.APPROVED).count()
+	total_to_tag = Package.query.filter_by(state=PackageState.APPROVED, tags=None).count()
 
-	total_topics = topic_query.count()
-	topics_to_add = topic_query \
-			.filter(~ db.exists().where(Package.forums==ForumTopic.topic_id)) \
-			.count()
+	unfulfilled_meta_packages = MetaPackage.query \
+			.filter(~ MetaPackage.packages.any(state=PackageState.APPROVED)) \
+			.filter(MetaPackage.dependencies.any(optional=False)) \
+			.order_by(db.asc(MetaPackage.name)).count()
 
-	return render_template("todo/list.html", title="Reports and Work Queue",
-		packages=packages, releases=releases, screenshots=screenshots,
-		canApproveNew=canApproveNew, canApproveRel=canApproveRel, canApproveScn=canApproveScn,
-		topics_to_add=topics_to_add, total_topics=total_topics)
+	return render_template("todo/editor.html", current_tab="editor",
+			packages=packages, wip_packages=wip_packages, releases=releases, screenshots=screenshots,
+			canApproveNew=canApproveNew, canApproveRel=canApproveRel, canApproveScn=canApproveScn,
+			total_packages=total_packages, total_to_tag=total_to_tag,
+			unfulfilled_meta_packages=unfulfilled_meta_packages)
 
 
 @bp.route("/todo/topics/")
@@ -89,14 +97,151 @@ def topics():
 		num = 100
 
 	query = query.paginate(page, num, True)
-	next_url = url_for("todo.topics", page=query.next_num, query=qb.search, \
-	 	show_discarded=qb.show_discarded, n=num, sort=qb.order_by) \
+	next_url = url_for("todo.topics", page=query.next_num, query=qb.search,
+			show_discarded=qb.show_discarded, n=num, sort=qb.order_by) \
 			if query.has_next else None
-	prev_url = url_for("todo.topics", page=query.prev_num, query=qb.search, \
-	 	show_discarded=qb.show_discarded, n=num, sort=qb.order_by) \
+	prev_url = url_for("todo.topics", page=query.prev_num, query=qb.search,
+			show_discarded=qb.show_discarded, n=num, sort=qb.order_by) \
 			if query.has_prev else None
 
-	return render_template("todo/topics.html", topics=query.items, total=total, \
-			topic_count=topic_count, query=qb.search, show_discarded=qb.show_discarded, \
-			next_url=next_url, prev_url=prev_url, page=page, page_max=query.pages, \
+	return render_template("todo/topics.html", current_tab="topics", topics=query.items, total=total,
+			topic_count=topic_count, query=qb.search, show_discarded=qb.show_discarded,
+			next_url=next_url, prev_url=prev_url, page=page, page_max=query.pages,
 			n=num, sort_by=qb.order_by)
+
+
+@bp.route("/todo/tags/")
+@login_required
+def tags():
+	qb    = QueryBuilder(request.args)
+	qb.setSortIfNone("score", "desc")
+	query = qb.buildPackageQuery()
+
+	tags = Tag.query.order_by(db.asc(Tag.title)).all()
+
+	return render_template("todo/tags.html", current_tab="tags", packages=query.all(), tags=tags)
+
+
+@bp.route("/user/tags/")
+def tags_user():
+	return redirect(url_for('todo.tags', author=current_user.username))
+
+
+@bp.route("/todo/metapackages/")
+@login_required
+def metapackages():
+	mpackages = MetaPackage.query \
+			.filter(~ MetaPackage.packages.any(state=PackageState.APPROVED)) \
+			.filter(MetaPackage.dependencies.any(optional=False)) \
+			.order_by(db.asc(MetaPackage.name)).all()
+
+	return render_template("todo/metapackages.html", mpackages=mpackages)
+
+
+@bp.route("/user/todo/")
+@bp.route("/users/<username>/todo/")
+@login_required
+def view_user(username=None):
+	if username is None:
+		return redirect(url_for("todo.view_user", username=current_user.username))
+
+	user : User = User.query.filter_by(username=username).first()
+	if not user:
+		abort(404)
+
+	if current_user != user and not current_user.rank.atLeast(UserRank.EDITOR):
+		abort(403)
+
+	unapproved_packages = user.packages \
+		.filter(or_(Package.state == PackageState.WIP,
+			Package.state == PackageState.CHANGES_NEEDED)) \
+		.order_by(db.asc(Package.created_at)).all()
+
+	outdated_packages = user.maintained_packages \
+			.filter(Package.state != PackageState.DELETED,
+					Package.update_config.has(PackageUpdateConfig.outdated_at.isnot(None))) \
+			.order_by(db.asc(Package.title)).all()
+
+	topics_to_add = ForumTopic.query \
+			.filter_by(author_id=user.id) \
+			.filter(~ db.exists().where(Package.forums == ForumTopic.topic_id)) \
+			.order_by(db.asc(ForumTopic.name), db.asc(ForumTopic.title)) \
+			.all()
+
+	needs_tags = user.maintained_packages \
+		.filter(Package.state != PackageState.DELETED) \
+		.filter_by(tags=None).order_by(db.asc(Package.title)).all()
+
+	return render_template("todo/user.html", current_tab="user", user=user,
+			unapproved_packages=unapproved_packages, outdated_packages=outdated_packages,
+			needs_tags=needs_tags, topics_to_add=topics_to_add)
+
+
+@bp.route("/users/<username>/update-configs/apply-all/", methods=["POST"])
+@login_required
+def apply_all_updates(username):
+	user: User = User.query.filter_by(username=username).first()
+	if not user:
+		abort(404)
+
+	if current_user != user and not current_user.rank.atLeast(UserRank.EDITOR):
+		abort(403)
+
+	outdated_packages = user.maintained_packages \
+		.filter(Package.state != PackageState.DELETED,
+			Package.update_config.has(PackageUpdateConfig.outdated_at.isnot(None))) \
+		.order_by(db.asc(Package.title)).all()
+
+	for package in outdated_packages:
+		if not package.checkPerm(current_user, Permission.MAKE_RELEASE):
+			continue
+
+		if package.releases.filter(or_(PackageRelease.task_id.isnot(None),
+				PackageRelease.commit_hash==package.update_config.last_commit)).count() > 0:
+			continue
+
+		title = package.update_config.get_title()
+		ref = package.update_config.get_ref()
+
+		rel = PackageRelease()
+		rel.package = package
+		rel.title = title
+		rel.url = ""
+		rel.task_id = uuid()
+		db.session.add(rel)
+		db.session.commit()
+
+		makeVCSRelease.apply_async((rel.id, ref),
+				task_id=rel.task_id)
+
+		msg = "Created release {} (Applied all Git Update Detection)".format(rel.title)
+		addNotification(package.maintainers, current_user, NotificationType.PACKAGE_EDIT, msg,
+				rel.getEditURL(), package)
+		addAuditLog(AuditSeverity.NORMAL, current_user, msg, package.getDetailsURL(), package)
+		db.session.commit()
+
+	return redirect(url_for("todo.view_user", username=username))
+
+
+@bp.route("/todo/outdated/")
+@login_required
+def outdated():
+	is_mtm_only = isYes(request.args.get("mtm"))
+
+	query = db.session.query(Package).select_from(PackageUpdateConfig) \
+			.filter(PackageUpdateConfig.outdated_at.isnot(None)) \
+			.join(PackageUpdateConfig.package) \
+			.filter(Package.state == PackageState.APPROVED)
+
+	if is_mtm_only:
+		query = query.filter(Package.repo.ilike("%github.com/minetest-mods/%"))
+
+	sort_by = request.args.get("sort")
+	if sort_by == "date":
+		query = query.order_by(db.desc(PackageUpdateConfig.outdated_at))
+	else:
+		sort_by = "score"
+		query = query.order_by(db.desc(Package.score))
+
+	return render_template("todo/outdated.html", current_tab="outdated",
+			outdated_packages=query.all(), sort_by=sort_by, is_mtm_only=is_mtm_only)

app/blueprints/users/__init__.py

@@ -2,4 +2,4 @@ from flask import Blueprint
 
 bp = Blueprint("users", __name__)
 
-from . import profile, claim
+from . import profile, claim, account, settings

app/blueprints/users/account.py

@@ -0,0 +1,386 @@
+# ContentDB
+# Copyright (C) 2020  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_login import current_user, login_required, logout_user, login_user
+from flask_wtf import FlaskForm
+from sqlalchemy import or_
+from wtforms import *
+from wtforms.validators import *
+
+from app.models import *
+from app.tasks.emails import send_verify_email, send_anon_email, send_unsubscribe_verify, send_user_email
+from app.utils import randomString, make_flask_login_password, is_safe_url, check_password_hash, addAuditLog, nonEmptyOrNone
+from passlib.pwd import genphrase
+
+from . import bp
+
+
+class LoginForm(FlaskForm):
+	username = StringField("Username or email", [InputRequired()])
+	password = PasswordField("Password", [InputRequired(), Length(6, 100)])
+	remember_me = BooleanField("Remember me", default=True)
+	submit = SubmitField("Sign in")
+
+
+def handle_login(form):
+	def show_safe_err(err):
+		if "@" in username:
+			flash("Incorrect email or password", "danger")
+		else:
+			flash(err, "danger")
+
+
+	username = form.username.data.strip()
+	user = User.query.filter(or_(User.username == username, User.email == username)).first()
+	if user is None:
+		return show_safe_err("User {} does not exist".format(username))
+
+	if not check_password_hash(user.password, form.password.data):
+		return show_safe_err("Incorrect password. Did you set one?")
+
+	if not user.is_active:
+		flash("You need to confirm the registration email", "danger")
+		return
+
+	addAuditLog(AuditSeverity.USER, user, "Logged in using password",
+			url_for("users.profile", username=user.username))
+	db.session.commit()
+
+	login_user(user, remember=form.remember_me.data)
+	flash("Logged in successfully.", "success")
+
+	next = request.args.get("next")
+	if next and not is_safe_url(next):
+		abort(400)
+
+	return redirect(next or url_for("homepage.home"))
+
+
+@bp.route("/user/login/", methods=["GET", "POST"])
+def login():
+	if current_user.is_authenticated:
+		next = request.args.get("next")
+		if next and not is_safe_url(next):
+			abort(400)
+
+		return redirect(next or url_for("homepage.home"))
+
+	form = LoginForm(request.form)
+	if form.validate_on_submit():
+		ret = handle_login(form)
+		if ret:
+			return ret
+
+	if request.method == "GET":
+		form.remember_me.data = True
+
+
+	return render_template("users/login.html", form=form)
+
+
+@bp.route("/user/logout/", methods=["GET", "POST"])
+def logout():
+	logout_user()
+	return redirect(url_for("homepage.home"))
+
+
+class RegisterForm(FlaskForm):
+	username = StringField("Username", [InputRequired()])
+	email = StringField("Email", [InputRequired(), Email()])
+	password = PasswordField("Password", [InputRequired(), Length(6, 100)])
+	submit = SubmitField("Register")
+
+
+def handle_register(form):
+	user_by_name = User.query.filter(or_(
+			User.username == form.username.data,
+			User.forums_username == form.username.data,
+			User.github_username == form.username.data)).first()
+	if user_by_name:
+		if user_by_name.rank == UserRank.NOT_JOINED and user_by_name.forums_username:
+			flash("An account already exists for that username but hasn't been claimed yet.", "danger")
+			return redirect(url_for("users.claim_forums", username=user_by_name.forums_username))
+		else:
+			flash("That username is already in use, please choose another.", "danger")
+			return
+
+
+	user_by_email = User.query.filter_by(email=form.email.data).first()
+	if user_by_email:
+		send_anon_email.delay(form.email.data, "Email already in use",
+				"We were unable to create the account as the email is already in use by {}. Try a different email address.".format(
+						user_by_email.display_name))
+		flash("Check your email address to verify your account", "success")
+		return redirect(url_for("homepage.home"))
+	elif EmailSubscription.query.filter_by(email=form.email.data, blacklisted=True).count() > 0:
+		flash("That email address has been unsubscribed/blacklisted, and cannot be used", "danger")
+		return
+
+	user = User(form.username.data, False, form.email.data, make_flask_login_password(form.password.data))
+	user.notification_preferences = UserNotificationPreferences(user)
+	db.session.add(user)
+
+	addAuditLog(AuditSeverity.USER, user, "Registered with email",
+			url_for("users.profile", username=user.username))
+
+	token = randomString(32)
+
+	ver = UserEmailVerification()
+	ver.user = user
+	ver.token = token
+	ver.email = form.email.data
+	db.session.add(ver)
+	db.session.commit()
+
+	send_verify_email.delay(form.email.data, token)
+
+	flash("Check your email address to verify your account", "success")
+	return redirect(url_for("homepage.home"))
+
+
+@bp.route("/user/register/", methods=["GET", "POST"])
+def register():
+	form = RegisterForm(request.form)
+	if form.validate_on_submit():
+		ret = handle_register(form)
+		if ret:
+			return ret
+
+	return render_template("users/register.html", form=form, suggested_password=genphrase(entropy=52, wordset="bip39"))
+
+
+class ForgotPasswordForm(FlaskForm):
+	email = StringField("Email", [InputRequired(), Email()])
+	submit = SubmitField("Reset Password")
+
+@bp.route("/user/forgot-password/", methods=["GET", "POST"])
+def forgot_password():
+	form = ForgotPasswordForm(request.form)
+	if form.validate_on_submit():
+		email = form.email.data
+		user = User.query.filter_by(email=email).first()
+		if user:
+			token = randomString(32)
+
+			addAuditLog(AuditSeverity.USER, user, "(Anonymous) requested a password reset",
+					url_for("users.profile", username=user.username), None)
+
+			ver = UserEmailVerification()
+			ver.user = user
+			ver.token = token
+			ver.email = email
+			ver.is_password_reset = True
+			db.session.add(ver)
+			db.session.commit()
+
+			send_verify_email.delay(form.email.data, token)
+		else:
+			send_anon_email.delay(email, "Unable to find account", """
+					<p>
+						We were unable to perform the password reset as we could not find an account
+						associated with this email.
+					</p>
+					<p>
+						If you weren't expecting to receive this email, then you can safely ignore it.
+					</p>
+			""")
+
+		flash("Check your email address to continue the reset", "success")
+		return redirect(url_for("homepage.home"))
+
+	return render_template("users/forgot_password.html", form=form)
+
+
+class SetPasswordForm(FlaskForm):
+	email = StringField("Email", [Optional(), Email()])
+	password = PasswordField("New password", [InputRequired(), Length(8, 100)])
+	password2 = PasswordField("Verify password", [InputRequired(), Length(8, 100), validators.EqualTo('password', message='Passwords must match')])
+	submit = SubmitField("Save")
+
+class ChangePasswordForm(FlaskForm):
+	old_password = PasswordField("Old password", [InputRequired(), Length(8, 100)])
+	password = PasswordField("New password", [InputRequired(), Length(8, 100)])
+	password2 = PasswordField("Verify password", [InputRequired(), Length(8, 100), validators.EqualTo('password', message='Passwords must match')])
+	submit = SubmitField("Save")
+
+
+def handle_set_password(form):
+	one = form.password.data
+	two = form.password2.data
+	if one != two:
+		flash("Passwords do not much", "danger")
+		return
+
+	addAuditLog(AuditSeverity.USER, current_user, "Changed their password", url_for("users.profile", username=current_user.username))
+
+	current_user.password = make_flask_login_password(form.password.data)
+
+	if hasattr(form, "email"):
+		newEmail = nonEmptyOrNone(form.email.data)
+		if newEmail and newEmail != current_user.email:
+			if EmailSubscription.query.filter_by(email=form.email.data, blacklisted=True).count() > 0:
+				flash("That email address has been unsubscribed/blacklisted, and cannot be used", "danger")
+				return
+
+			token = randomString(32)
+
+			ver = UserEmailVerification()
+			ver.user = current_user
+			ver.token = token
+			ver.email = newEmail
+			db.session.add(ver)
+
+	db.session.commit()
+	flash("Your password has been changed successfully.", "success")
+	return redirect(url_for("homepage.home"))
+
+
+@bp.route("/user/change-password/", methods=["GET", "POST"])
+@login_required
+def change_password():
+	form = ChangePasswordForm(request.form)
+
+	if form.validate_on_submit():
+		if check_password_hash(current_user.password, form.old_password.data):
+			ret = handle_set_password(form)
+			if ret:
+				return ret
+		else:
+			flash("Old password is incorrect", "danger")
+
+	return render_template("users/change_set_password.html", form=form,
+			suggested_password=genphrase(entropy=52, wordset="bip39"))
+
+
+@bp.route("/user/set-password/", methods=["GET", "POST"])
+@login_required
+def set_password():
+	if current_user.password:
+		return redirect(url_for("users.change_password"))
+
+	form = SetPasswordForm(request.form)
+	if current_user.email is None:
+		form.email.validators = [InputRequired(), Email()]
+
+	if form.validate_on_submit():
+		ret = handle_set_password(form)
+		if ret:
+			return ret
+
+	return render_template("users/change_set_password.html", form=form, optional=request.args.get("optional"),
+			suggested_password=genphrase(entropy=52, wordset="bip39"))
+
+
+@bp.route("/user/verify/")
+def verify_email():
+	token = request.args.get("token")
+	ver : UserEmailVerification = UserEmailVerification.query.filter_by(token=token).first()
+	if ver is None:
+		flash("Unknown verification token!", "danger")
+		return redirect(url_for("homepage.home"))
+
+	user = ver.user
+
+	addAuditLog(AuditSeverity.USER, user, "Confirmed their email",
+			url_for("users.profile", username=user.username))
+
+	was_activating = not user.is_active
+
+	if ver.email and user.email != ver.email:
+		if User.query.filter_by(email=ver.email).count() > 0:
+			flash("Another user is already using that email", "danger")
+			return redirect(url_for("homepage.home"))
+
+		flash("Confirmed email change", "success")
+
+		if user.email:
+			send_user_email.delay(user.email,
+					"Email address changed",
+					"Your email address has changed. If you didn't request this, please contact an administrator.")
+
+	user.is_active = True
+	user.email = ver.email
+
+	db.session.delete(ver)
+	db.session.commit()
+
+	if ver.is_password_reset:
+		login_user(user)
+		user.password = None
+		db.session.commit()
+
+		return redirect(url_for("users.set_password"))
+
+	if current_user.is_authenticated:
+		return redirect(url_for("users.profile", username=current_user.username))
+	elif was_activating:
+		flash("You may now log in", "success")
+		return redirect(url_for("users.login"))
+	else:
+		return redirect(url_for("homepage.home"))
+
+
+class UnsubscribeForm(FlaskForm):
+	email = StringField("Email", [InputRequired(), Email()])
+	submit = SubmitField("Send")
+
+
+def unsubscribe_verify():
+	form = UnsubscribeForm(request.form)
+	if form.validate_on_submit():
+		email = form.email.data
+		sub = EmailSubscription.query.filter_by(email=email).first()
+		if not sub:
+			sub = EmailSubscription(email)
+			db.session.add(sub)
+
+		sub.token = randomString(32)
+		db.session.commit()
+		send_unsubscribe_verify.delay(form.email.data)
+
+		flash("Check your email address to continue the unsubscribe", "success")
+		return redirect(url_for("homepage.home"))
+
+	return render_template("users/unsubscribe.html", form=form)
+
+
+def unsubscribe_manage(sub: EmailSubscription):
+	user = User.query.filter_by(email=sub.email).first()
+
+	if request.method == "POST":
+		if user:
+			user.email = None
+
+		sub.blacklisted = True
+		db.session.commit()
+
+		flash("That email is now blacklisted. Please contact an admin if you wish to undo this.", "success")
+		return redirect(url_for("homepage.home"))
+
+	return render_template("users/unsubscribe.html", user=user)
+
+
+@bp.route("/unsubscribe/", methods=["GET", "POST"])
+def unsubscribe():
+	token = request.args.get("token")
+	if token:
+		sub = EmailSubscription.query.filter_by(token=token).first()
+		if sub:
+			return unsubscribe_manage(sub)
+
+	return unsubscribe_verify()

app/blueprints/users/claim.py

@@ -1,98 +1,120 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
-
 from . import bp
 from flask import redirect, render_template, session, request, flash, url_for
-from flask_user import current_user
 from app.models import db, User, UserRank
-from app.utils import randomString, loginUser, rank_required
+from app.utils import randomString, login_user_set_active
 from app.tasks.forumtasks import checkForumAccount
-from app.tasks.phpbbparser import getProfile
+from app.utils.phpbbparser import getProfile
+import re
+
+
+def check_username(username):
+	return username is not None and len(username) >= 2 and re.match("^[A-Za-z0-9._-]*$", username)
+
+
 
 @bp.route("/user/claim/", methods=["GET", "POST"])
 def claim():
+	return render_template("users/claim.html")
+
+
+@bp.route("/user/claim-forums/", methods=["GET", "POST"])
+def claim_forums():
 	username = request.args.get("username")
 	if username is None:
 		username = ""
 	else:
 		method = request.args.get("method")
+
+		if not check_username(username):
+			flash("Invalid username - must only contain A-Za-z0-9._. Consider contacting an admin", "danger")
+			return redirect(url_for("users.claim_forums"))
+
 		user = User.query.filter_by(forums_username=username).first()
 		if user and user.rank.atLeast(UserRank.NEW_MEMBER):
 			flash("User has already been claimed", "danger")
-			return redirect(url_for("users.claim"))
-		elif user is None and method == "github":
-			flash("Unable to get Github username for user", "danger")
-			return redirect(url_for("users.claim"))
-		elif user is None:
-			flash("Unable to find that user", "danger")
-			return redirect(url_for("users.claim"))
+			return redirect(url_for("users.claim_forums"))
+		elif method == "github":
+			if user is None or user.github_username is None:
+				flash("Unable to get GitHub username for user", "danger")
+				return redirect(url_for("users.claim_forums", username=username))
+			else:
+				return redirect(url_for("github.start"))
 
-		if user is not None and method == "github":
-			return redirect(url_for("github.start"))
-
-	token = None
 	if "forum_token" in session:
 		token = session["forum_token"]
 	else:
-		token = randomString(32)
+		token = randomString(12)
 		session["forum_token"] = token
 
 	if request.method == "POST":
 		ctype	= request.form.get("claim_type")
 		username = request.form.get("username")
 
-		if username is None or len(username.strip()) < 2:
-			flash("Invalid username", "danger")
+		if not check_username(username):
+			flash("Invalid username - must only contain A-Za-z0-9._. Consider contacting an admin", "danger")
 		elif ctype == "github":
 			task = checkForumAccount.delay(username)
-			return redirect(url_for("tasks.check", id=task.id, r=url_for("users.claim", username=username, method="github")))
+			return redirect(url_for("tasks.check", id=task.id, r=url_for("users.claim_forums", username=username, method="github")))
 		elif ctype == "forum":
 			user = User.query.filter_by(forums_username=username).first()
 			if user is not None and user.rank.atLeast(UserRank.NEW_MEMBER):
 				flash("That user has already been claimed!", "danger")
-				return redirect(url_for("users.claim"))
+				return redirect(url_for("users.claim_forums"))
 
 			# Get signature
 			sig = None
 			try:
 				profile = getProfile("https://forum.minetest.net", username)
-				sig = profile.signature
-			except IOError:
+				sig = profile.signature if profile else None
+			except IOError as e:
+				if hasattr(e, 'message'):
+					message = e.message
+				else:
+					message = str(e)
+
+				flash("Error whilst attempting to access forums: " + message, "danger")
+				return redirect(url_for("users.claim_forums", username=username))
+
+			if profile is None:
 				flash("Unable to get forum signature - does the user exist?", "danger")
-				return redirect(url_for("users.claim", username=username))
+				return redirect(url_for("users.claim_forums", username=username))
 
 			# Look for key
-			if token in sig:
+			if sig and token in sig:
+				# Try getting again to fix crash
+				user = User.query.filter_by(forums_username=username).first()
 				if user is None:
 					user = User(username)
 					user.forums_username = username
 					db.session.add(user)
 					db.session.commit()
 
-				if loginUser(user):
+				if login_user_set_active(user, remember=True):
 					return redirect(url_for("users.set_password"))
 				else:
 					flash("Unable to login as user", "danger")
-					return redirect(url_for("users.claim", username=username))
+					return redirect(url_for("users.claim_forums", username=username))
 
 			else:
 				flash("Could not find the key in your signature!", "danger")
-				return redirect(url_for("users.claim", username=username))
+				return redirect(url_for("users.claim_forums", username=username))
 		else:
 			flash("Unknown claim type", "danger")
 
-	return render_template("users/claim.html", username=username, key=token)
+	return render_template("users/claim_forums.html", username=username, key="cdb_" + token)

app/blueprints/users/profile.py

@@ -1,117 +1,61 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
 from flask import *
-from flask_user import *
-from flask_login import login_user, logout_user
-from app.markdown import render_markdown
-from . import bp
-from app.models import *
-from flask_wtf import FlaskForm
-from wtforms import *
-from wtforms.validators import *
-from app.utils import randomString, loginUser, rank_required
-from app.tasks.forumtasks import checkForumAccount
-from app.tasks.emails import sendVerifyEmail, sendEmailRaw
-from app.tasks.phpbbparser import getProfile
+from flask_login import current_user, login_required
+from sqlalchemy import func
 
-# Define the User profile form
-class UserProfileForm(FlaskForm):
-	display_name = StringField("Display name", [Optional(), Length(2, 20)])
-	email = StringField("Email", [Optional(), Email()], filters = [lambda x: x or None])
-	website_url = StringField("Website URL", [Optional(), URL()], filters = [lambda x: x or None])
-	donate_url = StringField("Donation URL", [Optional(), URL()], filters = [lambda x: x or None])
-	rank = SelectField("Rank", [Optional()], choices=UserRank.choices(), coerce=UserRank.coerce, default=UserRank.NEW_MEMBER)
-	submit = SubmitField("Save")
+from app.models import *
+from app.tasks.forumtasks import checkForumAccount
+from . import bp
 
 
 @bp.route("/users/", methods=["GET"])
 def list_all():
-	users = User.query.order_by(db.desc(User.rank), db.asc(User.display_name)).all()
+	users = db.session.query(User, func.count(Package.id)) \
+			.select_from(User).outerjoin(Package) \
+			.order_by(db.desc(User.rank), db.asc(User.display_name)) \
+			.group_by(User.id).all()
+
 	return render_template("users/list.html", users=users)
 
 
-@bp.route("/users/<username>/", methods=["GET", "POST"])
+@bp.route("/user/forum/<username>/")
+def by_forums_username(username):
+	user = User.query.filter_by(forums_username=username).first()
+	if user:
+		return redirect(url_for("users.profile", username=user.username))
+
+	return render_template("users/forums_no_such_user.html", username=username)
+
+
+@bp.route("/users/<username>/")
 def profile(username):
 	user = User.query.filter_by(username=username).first()
 	if not user:
 		abort(404)
 
-	form = None
-	if user.checkPerm(current_user, Permission.CHANGE_DNAME) or \
-			user.checkPerm(current_user, Permission.CHANGE_EMAIL) or \
-			user.checkPerm(current_user, Permission.CHANGE_RANK):
-		# Initialize form
-		form = UserProfileForm(formdata=request.form, obj=user)
-
-		# Process valid POST
-		if request.method=="POST" and form.validate():
-			# Copy form fields to user_profile fields
-			if user.checkPerm(current_user, Permission.CHANGE_DNAME):
-				user.display_name = form["display_name"].data
-
-			if user.checkPerm(current_user, Permission.CHANGE_PROFILE_URLS):
-				user.website_url  = form["website_url"].data
-				user.donate_url   = form["donate_url"].data
-
-			if user.checkPerm(current_user, Permission.CHANGE_RANK):
-				newRank = form["rank"].data
-				if current_user.rank.atLeast(newRank):
-					user.rank = form["rank"].data
-				else:
-					flash("Can't promote a user to a rank higher than yourself!", "danger")
-
-			if user.checkPerm(current_user, Permission.CHANGE_EMAIL):
-				newEmail = form["email"].data
-				if newEmail != user.email and newEmail.strip() != "":
-					token = randomString(32)
-
-					ver = UserEmailVerification()
-					ver.user  = user
-					ver.token = token
-					ver.email = newEmail
-					db.session.add(ver)
-					db.session.commit()
-
-					task = sendVerifyEmail.delay(newEmail, token)
-					return redirect(url_for("tasks.check", id=task.id, r=url_for("users.profile", username=username)))
-
-			# Save user_profile
-			db.session.commit()
-
-			# Redirect to home page
-			return redirect(url_for("users.profile", username=username))
-
-	packages = user.packages.filter_by(soft_deleted=False)
+	packages = user.packages.filter(Package.state != PackageState.DELETED)
 	if not current_user.is_authenticated or (user != current_user and not current_user.canAccessTodoList()):
-		packages = packages.filter_by(approved=True)
+		packages = packages.filter_by(state=PackageState.APPROVED)
 	packages = packages.order_by(db.asc(Package.title))
 
-	topics_to_add = None
-	if current_user == user or user.checkPerm(current_user, Permission.CHANGE_AUTHOR):
-		topics_to_add = ForumTopic.query \
-					.filter_by(author_id=user.id) \
-					.filter(~ db.exists().where(Package.forums==ForumTopic.topic_id)) \
-					.order_by(db.asc(ForumTopic.name), db.asc(ForumTopic.title)) \
-					.all()
-
 	# Process GET or invalid POST
-	return render_template("users/profile.html",
-			user=user, form=form, packages=packages, topics_to_add=topics_to_add)
+	return render_template("users/profile.html", user=user, packages=packages)
 
 
 @bp.route("/users/<username>/check/", methods=["POST"])
@@ -131,108 +75,3 @@ def user_check(username):
 	next_url = url_for("users.profile", username=username)
 
 	return redirect(url_for("tasks.check", id=task.id, r=next_url))
-
-
-class SendEmailForm(FlaskForm):
-	subject = StringField("Subject", [InputRequired(), Length(1, 300)])
-	text    = TextAreaField("Message", [InputRequired()])
-	submit  = SubmitField("Send")
-
-
-@bp.route("/users/<username>/email/", methods=["GET", "POST"])
-@rank_required(UserRank.MODERATOR)
-def send_email(username):
-	user = User.query.filter_by(username=username).first()
-	if user is None:
-		abort(404)
-
-	next_url = url_for("users.profile", username=user.username)
-
-	if user.email is None:
-		flash("User has no email address!", "danger")
-		return redirect(next_url)
-
-	form = SendEmailForm(request.form)
-	if form.validate_on_submit():
-		text = form.text.data
-		html = render_markdown(text)
-		task = sendEmailRaw.delay([user.email], form.subject.data, text, html)
-		return redirect(url_for("tasks.check", id=task.id, r=next_url))
-
-	return render_template("users/send_email.html", form=form)
-
-
-
-class SetPasswordForm(FlaskForm):
-	email = StringField("Email", [Optional(), Email()])
-	password = PasswordField("New password", [InputRequired(), Length(2, 100)])
-	password2 = PasswordField("Verify password", [InputRequired(), Length(2, 100)])
-	submit = SubmitField("Save")
-
-@bp.route("/user/set-password/", methods=["GET", "POST"])
-@login_required
-def set_password():
-	if current_user.hasPassword():
-		return redirect(url_for("user.change_password"))
-
-	form = SetPasswordForm(request.form)
-	if current_user.email == None:
-		form.email.validators = [InputRequired(), Email()]
-
-	if request.method == "POST" and form.validate():
-		one = form.password.data
-		two = form.password2.data
-		if one == two:
-			# Hash password
-			hashed_password = user_manager.hash_password(form.password.data)
-
-			# Change password
-			current_user.password = hashed_password
-			db.session.commit()
-
-			# Send 'password_changed' email
-			if user_manager.USER_ENABLE_EMAIL and current_user.email:
-				emails.send_password_changed_email(current_user)
-
-			# Send password_changed signal
-			signals.user_changed_password.send(current_app._get_current_object(), user=current_user)
-
-			# Prepare one-time system message
-			flash('Your password has been changed successfully.', 'success')
-
-			newEmail = form["email"].data
-			if newEmail != current_user.email and newEmail.strip() != "":
-				token = randomString(32)
-
-				ver = UserEmailVerification()
-				ver.user = current_user
-				ver.token = token
-				ver.email = newEmail
-				db.session.add(ver)
-				db.session.commit()
-
-				task = sendVerifyEmail.delay(newEmail, token)
-				return redirect(url_for("tasks.check", id=task.id, r=url_for("users.profile", username=current_user.username)))
-			else:
-				return redirect(url_for("user.login"))
-		else:
-			flash("Passwords do not match", "danger")
-
-	return render_template("users/set_password.html", form=form, optional=request.args.get("optional"))
-
-
-@bp.route("/users/verify/")
-def verify_email():
-	token = request.args.get("token")
-	ver = UserEmailVerification.query.filter_by(token=token).first()
-	if ver is None:
-		flash("Unknown verification token!", "danger")
-	else:
-		ver.user.email = ver.email
-		db.session.delete(ver)
-		db.session.commit()
-
-	if current_user.is_authenticated:
-		return redirect(url_for("users.profile", username=current_user.username))
-	else:
-		return redirect(url_for("homepage.home"))

app/blueprints/users/settings.py

@@ -0,0 +1,257 @@
+from flask import *
+from flask_login import current_user, login_required, logout_user
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+
+from app.models import *
+from app.utils import nonEmptyOrNone, addAuditLog, randomString, rank_required
+from app.tasks.emails import send_verify_email
+from . import bp
+
+
+def get_setting_tabs(user):
+	return [
+		{
+			"id": "edit_profile",
+			"title": "Edit Profile",
+			"url": url_for("users.profile_edit", username=user.username)
+		},
+		{
+			"id": "account",
+			"title": "Account and Security",
+			"url": url_for("users.account", username=user.username)
+		},
+		{
+			"id": "notifications",
+			"title": "Email and Notifications",
+			"url": url_for("users.email_notifications", username=user.username)
+		},
+		{
+			"id": "api_tokens",
+			"title": "API Tokens",
+			"url": url_for("api.list_tokens", username=user.username)
+		},
+	]
+
+
+class UserProfileForm(FlaskForm):
+	website_url = StringField("Website URL", [Optional(), URL()], filters = [lambda x: x or None])
+	donate_url = StringField("Donation URL", [Optional(), URL()], filters = [lambda x: x or None])
+	submit = SubmitField("Save")
+
+
+@bp.route("/users/<username>/settings/profile/", methods=["GET", "POST"])
+@login_required
+def profile_edit(username):
+	user : User = User.query.filter_by(username=username).first()
+	if not user:
+		abort(404)
+
+	if not user.can_see_edit_profile(current_user):
+		flash("Permission denied", "danger")
+		return redirect(url_for("users.profile", username=username))
+
+	form = UserProfileForm(obj=user)
+	if form.validate_on_submit():
+		severity = AuditSeverity.NORMAL if current_user == user else AuditSeverity.MODERATION
+		addAuditLog(severity, current_user, "Edited {}'s profile".format(user.display_name),
+				url_for("users.profile", username=username))
+
+		if user.checkPerm(current_user, Permission.CHANGE_PROFILE_URLS):
+			user.website_url  = form["website_url"].data
+			user.donate_url   = form["donate_url"].data
+
+		db.session.commit()
+
+		return redirect(url_for("users.profile", username=username))
+
+	# Process GET or invalid POST
+	return render_template("users/profile_edit.html", user=user, form=form, tabs=get_setting_tabs(user), current_tab="edit_profile")
+
+
+def make_settings_form():
+	attrs = {
+		"email": StringField("Email", [Optional(), Email()]),
+		"submit": SubmitField("Save")
+	}
+
+	for notificationType in NotificationType:
+		key = "pref_" + notificationType.toName()
+		attrs[key] = BooleanField("")
+		attrs[key + "_digest"] = BooleanField("")
+
+	return type("SettingsForm", (FlaskForm,), attrs)
+
+SettingsForm = make_settings_form()
+
+
+def handle_email_notifications(user, prefs: UserNotificationPreferences, is_new, form):
+	for notificationType in NotificationType:
+		field_email = getattr(form, "pref_" + notificationType.toName()).data
+		field_digest = getattr(form, "pref_" + notificationType.toName() + "_digest").data or field_email
+		prefs.set_can_email(notificationType, field_email)
+		prefs.set_can_digest(notificationType, field_digest)
+
+	if is_new:
+		db.session.add(prefs)
+
+	if user.checkPerm(current_user, Permission.CHANGE_EMAIL):
+		newEmail = form.email.data
+		if newEmail and newEmail != user.email and newEmail.strip() != "":
+			if EmailSubscription.query.filter_by(email=form.email.data, blacklisted=True).count() > 0:
+				flash("That email address has been unsubscribed/blacklisted, and cannot be used", "danger")
+				return
+
+			token = randomString(32)
+
+			severity = AuditSeverity.NORMAL if current_user == user else AuditSeverity.MODERATION
+
+			msg = "Changed email of {}".format(user.display_name)
+			addAuditLog(severity, current_user, msg, url_for("users.profile", username=user.username))
+
+			ver = UserEmailVerification()
+			ver.user = user
+			ver.token = token
+			ver.email = newEmail
+			db.session.add(ver)
+			db.session.commit()
+
+			flash("Check your email to confirm it", "success")
+
+			send_verify_email.delay(newEmail, token)
+			return redirect(url_for("users.email_notifications", username=user.username))
+
+	db.session.commit()
+	return redirect(url_for("users.email_notifications", username=user.username))
+
+
+@bp.route("/user/settings/email/")
+@bp.route("/users/<username>/settings/email/", methods=["GET", "POST"])
+@login_required
+def email_notifications(username=None):
+	if username is None:
+		return redirect(url_for("users.email_notifications", username=current_user.username))
+
+	user: User = User.query.filter_by(username=username).first()
+	if not user:
+		abort(404)
+
+	if not user.checkPerm(current_user, Permission.CHANGE_EMAIL):
+		abort(403)
+
+	is_new = False
+	prefs = user.notification_preferences
+	if prefs is None:
+		is_new = True
+		prefs = UserNotificationPreferences(user)
+
+	data = {}
+	types = []
+	for notificationType in NotificationType:
+		types.append(notificationType)
+		data["pref_" + notificationType.toName()] = prefs.get_can_email(notificationType)
+		data["pref_" + notificationType.toName() + "_digest"] = prefs.get_can_digest(notificationType)
+
+	data["email"] = user.email
+
+	form = SettingsForm(data=data)
+	if form.validate_on_submit():
+		ret = handle_email_notifications(user, prefs, is_new, form)
+		if ret:
+			return ret
+
+	return render_template("users/settings_email.html",
+			form=form, user=user, types=types, is_new=is_new,
+			tabs=get_setting_tabs(user), current_tab="notifications")
+
+
+class UserAccountForm(FlaskForm):
+	display_name = StringField("Display name", [Optional(), Length(2, 100)])
+	forums_username = StringField("Forums Username", [Optional(), Length(2, 50)])
+	github_username = StringField("GitHub Username", [Optional(), Length(2, 50)])
+	rank = SelectField("Rank", [Optional()], choices=UserRank.choices(), coerce=UserRank.coerce,
+			default=UserRank.NEW_MEMBER)
+	submit = SubmitField("Save")
+
+
+@bp.route("/users/<username>/settings/account/", methods=["GET", "POST"])
+@login_required
+def account(username):
+	user : User = User.query.filter_by(username=username).first()
+	if not user:
+		abort(404)
+
+	if not user.can_see_edit_profile(current_user):
+		flash("Permission denied", "danger")
+		return redirect(url_for("users.profile", username=username))
+
+	can_edit_account_settings = user.checkPerm(current_user, Permission.CHANGE_USERNAMES) or \
+			user.checkPerm(current_user, Permission.CHANGE_RANK)
+	form = UserAccountForm(obj=user) if can_edit_account_settings else None
+	if form and form.validate_on_submit():
+		severity = AuditSeverity.NORMAL if current_user == user else AuditSeverity.MODERATION
+		addAuditLog(severity, current_user, "Edited {}'s profile".format(user.display_name),
+				url_for("users.profile", username=username))
+
+		# Copy form fields to user_profile fields
+		if user.checkPerm(current_user, Permission.CHANGE_USERNAMES):
+			user.display_name = form.display_name.data
+			user.forums_username = nonEmptyOrNone(form.forums_username.data)
+			user.github_username = nonEmptyOrNone(form.github_username.data)
+
+		if user.checkPerm(current_user, Permission.CHANGE_RANK):
+			newRank = form["rank"].data
+			if current_user.rank.atLeast(newRank):
+				if newRank != user.rank:
+					user.rank = form["rank"].data
+					msg = "Set rank of {} to {}".format(user.display_name, user.rank.getTitle())
+					addAuditLog(AuditSeverity.MODERATION, current_user, msg,
+							url_for("users.profile", username=username))
+			else:
+				flash("Can't promote a user to a rank higher than yourself!", "danger")
+
+		db.session.commit()
+
+		return redirect(url_for("users.account", username=username))
+
+	return render_template("users/account.html", user=user, form=form, tabs=get_setting_tabs(user), current_tab="account")
+
+
+@bp.route("/users/<username>/delete/", methods=["GET", "POST"])
+@rank_required(UserRank.ADMIN)
+def delete(username):
+	user: User = User.query.filter_by(username=username).first()
+	if not user:
+		abort(404)
+
+	if user.rank.atLeast(UserRank.MODERATOR):
+		flash("Users with moderator rank or above cannot be deleted", "danger")
+		return redirect(url_for("users.account", username=username))
+
+	if request.method == "GET":
+		return render_template("users/delete.html", user=user, can_delete=user.can_delete())
+
+	if user.can_delete():
+		msg = "Deleted user {}".format(user.username)
+		flash(msg, "success")
+		addAuditLog(AuditSeverity.MODERATION, current_user, msg, None)
+
+		db.session.delete(user)
+	else:
+		user.replies.delete()
+		for thread in user.threads.all():
+			db.session.delete(thread)
+		user.email = None
+		user.rank = UserRank.NOT_JOINED
+
+		msg = "Deactivated user {}".format(user.username)
+		flash(msg, "success")
+		addAuditLog(AuditSeverity.MODERATION, current_user, msg, None)
+
+	db.session.commit()
+
+	if user == current_user:
+		logout_user()
+
+	return redirect(url_for("homepage.home"))

app/default_data.py

@@ -1,11 +1,11 @@
 from .models import *
-from .utils import make_flask_user_password
+from .utils import make_flask_login_password
 
 
 def populate(session):
 	admin_user = User("rubenwardy")
-	admin_user.active = True
-	admin_user.password = make_flask_user_password("tuckfrump")
+	admin_user.is_active = True
+	admin_user.password = make_flask_login_password("tuckfrump")
 	admin_user.github_username = "rubenwardy"
 	admin_user.forums_username = "rubenwardy"
 	admin_user.rank = UserRank.ADMIN
@@ -17,10 +17,10 @@ def populate(session):
 	session.add(MinetestRelease("5.1", 38))
 
 	tags = {}
-	for tag in ["Inventory", "Mapgen", "Building", \
-			"Mobs and NPCs", "Tools", "Player effects", \
-			"Environment", "Transport", "Maintenance", "Plants and farming", \
-			"PvP", "PvE", "Survival", "Creative", "Puzzle", "Multiplayer", "Singleplayer"]:
+	for tag in ["Inventory", "Mapgen", "Building",
+		"Mobs and NPCs", "Tools", "Player effects",
+		"Environment", "Transport", "Maintenance", "Plants and farming",
+		"PvP", "PvE", "Survival", "Creative", "Puzzle", "Multiplayer", "Singleplayer"]:
 		row = Tag(tag)
 		tags[row.name] = row
 		session.add(row)
@@ -63,7 +63,7 @@ def populate_test_data(session):
 
 
 	mod = Package()
-	mod.approved = True
+	mod.state = PackageState.APPROVED
 	mod.name = "alpha"
 	mod.title = "Alpha Test"
 	mod.license = licenses["MIT"]
@@ -87,7 +87,7 @@ def populate_test_data(session):
 	session.add(rel)
 
 	mod1 = Package()
-	mod1.approved = True
+	mod1.state = PackageState.APPROVED
 	mod1.name = "awards"
 	mod1.title = "Awards"
 	mod1.license = licenses["LGPLv2.1"]
@@ -102,7 +102,7 @@ def populate_test_data(session):
 	mod1.desc = """
 Majority of awards are back ported from Calinou's old fork in Carbone, under same license.
 
-```
+```lua
 awards.register_achievement("award_mesefind",{
     title = "First Mese Find",
     description = "Found some Mese!",
@@ -124,7 +124,7 @@ awards.register_achievement("award_mesefind",{
 	session.add(rel)
 
 	mod2 = Package()
-	mod2.approved = True
+	mod2.state = PackageState.APPROVED
 	mod2.name = "mesecons"
 	mod2.title = "Mesecons"
 	mod2.tags.append(tags["tools"])
@@ -213,7 +213,7 @@ No warranty is provided, express or implied, for any part of the project.
 	session.add(mod2)
 
 	mod = Package()
-	mod.approved = True
+	mod.state = PackageState.APPROVED
 	mod.name = "handholds"
 	mod.title = "Handholds"
 	mod.license = licenses["MIT"]
@@ -237,7 +237,7 @@ No warranty is provided, express or implied, for any part of the project.
 	session.add(rel)
 
 	mod = Package()
-	mod.approved = True
+	mod.state = PackageState.APPROVED
 	mod.name = "other_worlds"
 	mod.title = "Other Worlds"
 	mod.license = licenses["MIT"]
@@ -254,7 +254,7 @@ No warranty is provided, express or implied, for any part of the project.
 	session.add(mod)
 
 	mod = Package()
-	mod.approved = True
+	mod.state = PackageState.APPROVED
 	mod.name = "food"
 	mod.title = "Food"
 	mod.license = licenses["LGPLv2.1"]
@@ -270,7 +270,7 @@ No warranty is provided, express or implied, for any part of the project.
 	session.add(mod)
 
 	mod = Package()
-	mod.approved = True
+	mod.state = PackageState.APPROVED
 	mod.name = "food_sweet"
 	mod.title = "Sweet Foods"
 	mod.license = licenses["CC0"]
@@ -287,7 +287,7 @@ No warranty is provided, express or implied, for any part of the project.
 	session.add(mod)
 
 	game1 = Package()
-	game1.approved = True
+	game1.state = PackageState.APPROVED
 	game1.name = "capturetheflag"
 	game1.title = "Capture The Flag"
 	game1.type = PackageType.GAME
@@ -350,7 +350,7 @@ Uses the CTF PvP Engine.
 
 
 	mod = Package()
-	mod.approved = True
+	mod.state = PackageState.APPROVED
 	mod.name = "pixelbox"
 	mod.title = "PixelBOX Reloaded"
 	mod.license = licenses["CC0"]

app/flatpages/help.md

@@ -1,8 +1,27 @@
 title: Help
+toc: False
+
+## General Help
 
-* [Package Tags](package_tags)
-* [Ranks and Permissions](ranks_permissions)
 * [Content Ratings and Flags](content_flags)
+* [Non-free Licenses](non_free)
+* [Why WTFPL is a terrible license](wtfpl)
+* [Ranks and Permissions](ranks_permissions)
 * [Reporting Content](reporting)
-* [API](api)
+* [Top Packages Algorithm](top_packages)
+
+## Help for Package Authors
+
+* [Package Inclusion Policy and Guidance](/policy_and_guidance/)
+* [Git Update Detection](update_config)
 * [Creating Releases using Webhooks](release_webhooks)
+* [Package Configuration and Releases Guide](package_config)
+
+## Help for Specific User Ranks
+
+* [Editors](editors)
+
+## APIs
+
+* [API](api)
+* [Prometheus Metrics](metrics)

app/flatpages/help/api.md

@@ -1,64 +1,230 @@
 title: API
 
+## Responses and Error Handling
+
+If there is an error, the response will be JSON similar to the following with a non-200 status code: 
+
+```json
+{
+    "success": false,
+    "error": "The error message"
+}
+```
+
+Successful GET requests will return the resource's information directly as a JSON response.
+
+Other successful results will return a dictionary with `success` equaling true, and
+often other keys with information.
+
+
 ## Authentication
 
-Not all endpoints require authentication.
-Authentication is done using Bearer tokens:
+Not all endpoints require authentication, but it is done using Bearer tokens:
 
-	Authorization: Bearer YOURTOKEN
+```bash
+curl https://content.minetest.net/api/whoami/ \
+    -H "Authorization: Bearer YOURTOKEN" 
+```
 
-You can use the `/api/whoami` to check authentication.
+Tokens can be attained by visiting [Settings > API Tokens](/user/tokens/).
 
-## Endpoints
-
-### Misc
-
-* GET `/api/whoami/` - Json dictionary with the following keys:
-	* `is_authenticated` - True on successful API authentication
-	* `username` - Username of the user authenticated as, null otherwise.
-	* 403 will be thrown on unsupported authentication type, invalid access token, or other errors.
-
-### Packages
-
-* GET `/api/packages/` - See [Package Queries](#package-queries)
-* GET `/api/packages/<username>/<name>/`
-
-### Releases
-
-* GET `/api/packages/<username>/<name>/releases/`
-* POST `/api/packages/<username>/<name>/releases/new/`
-	* Requires authentication.
-	* `title`: human-readable name of the release.
-	* `method`: Release-creation method, only `git` is supported.
-	* `min_protocol`: (Optional) minimum Minetest protocol version. See [Minetest](#minetest).
-	* `min_protocol`: (Optional) maximum Minetest protocol version. See [Minetest](#minetest).
-	* If `git` release-creation method:
-		* `ref` - git reference, eg: `master`.
+* GET `/api/whoami/`:  JSON dictionary with the following keys:
+    * `is_authenticated`:  True on successful API authentication
+    * `username`:  Username of the user authenticated as, null otherwise.
+    * 4xx status codes will be thrown on unsupported authentication type, invalid access token, or other errors.
 
 
-### Topics
+## Packages
 
-* GET `/api/topics/` - Supports [Package Queries](#package-queries), and the following two options:
-    * `show_added` - Show topics which exist as packages, default true.
-	* `show_discarded` - Show topics which have been marked as outdated, default false.
+* GET `/api/packages/` (List)
+    * See [Package Queries](#package-queries)
+* GET `/api/packages/<username>/<name>/` (Read)
+* PUT `/api/packages/<author>/<name>/` (Update)
+    * Requires authentication.
+    * JSON dictionary with any of these keys (all are optional, null to delete Nullables):
+        * `type`: One of `GAME`, `MOD`, `TXP`.
+        * `title`: Human-readable title.
+        * `name`: Technical name (needs permission if already approved).
+        * `short_description`
+        * `tags`: List of tag names, see [misc](#misc).
+        * `content_warnings`: List of content warning names, see [misc](#misc).
+        * `license`: A license name.
+        * `media_license`: A license name.          
+        * `long_description`: Long markdown description.
+        * `repo`: Git repo URL.
+        * `website`: Website URL.
+        * `issue_tracker`: Issue tracker URL.
+        * `forums`: forum topic ID.
+* GET `/api/packages/<username>/<name>/dependencies/`
+    * If query argument `only_hard` is present, only hard deps will be returned.
 
-### Minetest
+Examples:
 
-* GET `/api/minetest_versions/`
+```bash
+# Edit packages
+curl -X PUT http://localhost:5123/api/packages/username/name/ \
+    -H "Authorization: Bearer YOURTOKEN" -H "Content-Type: application/json" \
+    -d '{ "title": "Foo bar", "tags": ["pvp", "survival"], "license": "MIT" }'
+    
+# Remove website URL
+curl -X PUT http://localhost:5123/api/packages/username/name/ \
+    -H "Authorization: Bearer YOURTOKEN" -H "Content-Type: application/json" \
+    -d '{ "website": null }'
+```
 
-
-## Package Queries
+### Package Queries
 
 Example:
 
-	/api/packages/?type=mod&type=game&q=mobs+fun&hide=nonfree&hide=gore
+    /api/packages/?type=mod&type=game&q=mobs+fun&hide=nonfree&hide=gore
 
 Supported query parameters:
 
-* `type` - Package types (`mod`, `game`, `txp`).
-* `q` - Query string
-* `random` - When present, enable random ordering and ignore `sort`.
-* `hide` - Hide content based on [Content Flags](content_flags).
-* `sort` - Sort by (`name`, `views`, `date`, `score`).
-* `order` - Sort ascending (`Asc`) or descending (`desc`).
-* `protocol_version` - Only show packages supported by this Minetest protocol version.
+* `type`:  Package types (`mod`, `game`, `txp`).
+* `q`:  Query string.
+* `author`:  Filter by author.
+* `tag`:  Filter by tags.
+* `random`:  When present, enable random ordering and ignore `sort`.
+* `limit`:  Return at most `limit` packages.
+* `hide`:  Hide content based on [Content Flags](/help/content_flags/).
+* `sort`:  Sort by (`name`, `title`, `score`, `reviews`, `downloads`, `created_at`, `approved_at`, `last_release`).
+* `order`:  Sort ascending (`asc`) or descending (`desc`).
+* `protocol_version`:  Only show packages supported by this Minetest protocol version.
+* `engine_version`:  Only show packages supported by this Minetest engine version, eg: `5.3.0`.
+* `fmt`:  How the response is formated.
+    * `keys`:  author/name only.
+    * `short`:  stuff needed for the Minetest client. 
+
+
+## Releases
+
+* GET `/api/packages/<username>/<name>/releases/` (List)
+    * Returns array of release dictionaries with keys:
+        * `id`: release ID
+        * `title`: human-readable title
+        * `release_date`: Date released
+        * `url`: download URL
+        * `commit`: commit hash or null
+        * `downloads`: number of downloads
+        * `min_minetest_version`: dict or null, minimum supported minetest version (inclusive).
+        * `max_minetest_version`: dict or null, minimum supported minetest version (inclusive).
+* GET `/api/packages/<username>/<name>/releases/<id>/` (Read)
+* POST `/api/packages/<username>/<name>/releases/new/` (Create)
+    * Requires authentication.
+    * Body can be JSON or multipart form data. Zip uploads must be multipart form data.
+    * `title`: human-readable name of the release.
+    * For Git release creation:
+        * `method`: must be `git`.
+        * `ref`: (Optional) git reference, eg: `master`.
+    * For zip upload release creation: 
+        * `file`: multipart file to upload, like `<input type=file>`.
+    * You can set min and max Minetest Versions [using the content's .conf file](/help/package_config/).
+* DELETE `/api/packages/<username>/<name>/releases/<id>/` (Delete)
+    * Requires authentication.
+    * Deletes release.
+  
+Examples:
+
+```bash
+# Create release from Git
+curl -X POST https://content.minetest.net/api/packages/username/name/releases/new/ \
+    -H "Authorization: Bearer YOURTOKEN" -H "Content-Type: application/json" \
+    -d '{ "method": "git", "title": "My Release", "ref": "master" }'
+
+# Create release from zip upload
+curl -X POST https://content.minetest.net/api/packages/username/name/releases/new/ \
+    -H "Authorization: Bearer YOURTOKEN" \
+    -F title="My Release" -F file=@path/to/file.zip
+
+# Delete release
+curl -X DELETE https://content.minetest.net/api/packages/username/name/releases/3/ \
+    -H "Authorization: Bearer YOURTOKEN" 
+```
+
+
+## Screenshots
+
+* GET `/api/packages/<username>/<name>/screenshots/` (List)
+    * Returns array of screenshot dictionaries with keys:
+        * `id`: screenshot ID
+        * `approved`: true if approved and visible.
+        * `title`: human-readable name for the screenshot, shown as a caption and alt text.
+        * `url`: absolute URL to screenshot.
+        * `created_at`: ISO time.
+        * `order`: Number used in ordering.
+* GET `/api/packages/<username>/<name>/screenshots/<id>/` (Read)
+    * Returns screenshot dictionary like above.
+* POST `/api/packages/<username>/<name>/screenshots/new/` (Create)
+    * Requires authentication.
+    * Body is multipart form data.
+    * `title`: human-readable name for the screenshot, shown as a caption and alt text.
+    * `file`: multipart file to upload, like `<input type=file>`.
+* DELETE `/api/packages/<username>/<name>/screenshots/<id>/` (Delete)
+    * Requires authentication.
+    * Deletes screenshot.
+* POST `/api/packages/<username>/<name>/screenshots/order/`
+    * Requires authentication.
+    * Body is a JSON array containing the screenshot IDs in their order.
+
+Examples:
+
+```bash
+# Create screenshots
+curl -X POST https://content.minetest.net/api/packages/username/name/screenshots/new/ \
+    -H "Authorization: Bearer YOURTOKEN" \
+    -F title="My Release" -F file=@path/to/screnshot.png
+
+# Delete screenshot
+curl -X DELETE https://content.minetest.net/api/packages/username/name/screenshots/3/ \
+    -H "Authorization: Bearer YOURTOKEN" 
+    
+# Reorder screenshots
+curl -X POST https://content.minetest.net/api/packages/username/name/screenshots/order/ \
+    -H "Authorization: Bearer YOURTOKEN" -H "Content-Type: application/json" \
+    -d "[13, 2, 5, 7]"
+```
+
+
+## Topics
+
+* GET `/api/topics/`:  Supports [Package Queries](#package-queries), and the following two options:
+    * `show_added`:  Show topics which exist as packages, default true.
+    * `show_discarded`:  Show topics which have been marked as outdated, default false.
+
+### Topic Queries
+
+Example:
+
+    /api/topics/?q=mobs
+
+Supported query parameters:
+
+* `q`:  Query string.
+* `sort`:  Sort by (`name`, `views`, `date`).
+* `order`:  Sort ascending (`asc`) or descending (`desc`).
+* `show_added`:  Show topics that have an existing package.
+* `show_discarded`:  Show topics marked as discarded.
+* `limit`:  Return at most `limit` topics.
+
+
+## Misc
+
+* GET `/api/scores/`
+    * See [Package Queries](#package-queries)
+* GET `/api/tags/`:  List of:
+    * `name`:  technical name
+    * `title`:  human-readable title
+    * `description`:  tag description or null
+* GET `/api/licenses/`:  List of:
+    * `name`
+    * `is_foss`: whether the license is foss
+* GET `/api/homepage/`
+    * `count`:  number of packages
+    * `downloads`:  get number of downloads
+    * `new`:  new packages
+    * `updated`:  recently updated packages
+    * `pop_mod`:  popular mods
+    * `pop_txp`:  popular textures
+    * `pop_game`:  popular games
+    * `high_reviewed`:  highest reviewed
+* GET `/api/minetest_versions/`

app/flatpages/help/content_flags.md

@@ -6,21 +6,29 @@ your client to use new flags.
 
 ## Flags
 
+Minetest allows you to specify a comma-separated list of flags to hide in the
+client:
+
+```
+contentdb_flag_blacklist = nonfree, bad_language, drugs
+```
+
+A flag can be:
+
 * `nonfree` - can be used to hide packages which do not qualify as
 	'free software', as defined by the Free Software Foundation.
-* A content rating, given below.
+* A content warning, given below.
+* `android_default` - meta-flag that filters out any content with a content warning.
+* `desktop_default` - meta-flag that doesn't filter anything out for now.
 
+## Content Warnings
 
-## Ratings
+Packages with mature content will be tagged with a content warning based
+on the content type.
 
-Content ratings aren't currently supported by ContentDB.
-Instead, mature content isn't allowed at all for now.
-
-In the future, more mature content will be allowed but labelled with
-content ratings which may contain the following:
-
-* android_default - meta-rating which includes gore and drugs.
-* desktop_default - meta-rating which won't include anything for now.
-* gore - more than just blood
-* drugs
-* swearing
+* `bad_language` - swearing.
+* `drugs` - drugs or alcohol.
+* `gambling`
+* `gore` - blood, etc.
+* `horror` - shocking and scary content.
+* `violence` - non-cartoon violence.

app/flatpages/help/editors.md

@@ -0,0 +1,34 @@
+title: Editors
+
+## What should editors do?
+
+Editors are users of rank Editor or above.
+They are responsible for ensuring that the package listings of ContentDB are useful.
+For this purpose, they can/will:
+
+* Review and approve packages.
+* Edit any package - including tags, releases, screenshots, and maintainers.
+* Create packages on behalf of authors who aren't present.
+
+Editors should make sure they are familiar with the
+[Package Inclusion Policy and Guidance](/policy_and_guidance/).
+
+## ContentDB is not a curated platform
+
+It's important to note that ContentDB isn't a curated platform, but it also does have some
+requirements on minimum usefulness. See 2.2 in the [Policy and Guidance](/policy_and_guidance/).
+
+## Editor Work Queue
+
+The [Editor Work Queue](/todo/) and related pages contain useful information for editors, such as:
+
+* The package, release, and screenshot approval queues.
+* Packages which are outdated or are missing tags.
+* A list of forum topics without packages.
+  Editors can create the packages or "discard" them if they don't think it's worth adding them.
+
+## Editor Notifications
+
+Editors currently receive notifications for any new thread opened on a package, so that they
+know when a user is asking for help. These notifications are shown separately in the notifications
+interface, and can be configured separately in Emails and Notifications.

app/flatpages/help/metrics.md

@@ -0,0 +1,16 @@
+title: Prometheus Metrics
+
+## What is Prometheus?
+
+[Prometheus](https://prometheus.io) is an "open-source monitoring system with a
+dimensional data model, flexible query language, efficient time series database
+and modern alerting approach".
+
+Prometheus Metrics can be accessed at [/metrics](/metrics).
+
+## Metrics
+
+* `contentdb_packages` - Total packages (counter).
+* `contentdb_users` - Number of registered users (counter).
+* `contentdb_downloads` - Total downloads (counter).
+* `contentdb_score` - Total package score (gauge).

app/flatpages/help/non_free.md

@@ -0,0 +1,73 @@
+title: Non-free Licenses
+
+## What are Non-Free, Free, and Open Source licenses?
+
+A non-free license is one that does not meet the
+[Free Software Definition](https://www.gnu.org/philosophy/free-sw.en.html)
+or the [Open Source Definition](https://opensource.org/osd).
+ContentDB will clearly label any packages with non-free licenses,
+and they will be subject to limited promotion.
+
+## How does ContentDB deal with Non-Free Licenses?
+
+**ContentDB does not allow certain non-free licenses, and will limit the promotion
+of packages with non-free licenses.**
+
+Minetest is free and open source software, and is only as big as it is now
+because of this. It's pretty amazing you can take nearly any published mod and modify it
+to how you like - add some features, maybe fix some bugs - and then share those
+modifications without the worry of legal issues. The project, itself, relies on open
+source contributions to survive - if it were non-free, then it would have died
+when celeron55 lost interest.
+
+If you have played nearly any game with a large modding scene, you will find
+that most mods are legally ambiguous. A lot of them don't even provide the
+source code to allow you to bug fix or extend as you need.
+
+Limiting the promotion of problematic licenses helps Minetest avoid ending up in
+such a state. Licenses that prohibit redistribution or modification are
+completely banned from ContentDB and the Minetest forums. Other non-free licenses
+will be subject to limited promotion - they won't be shown by default in
+the client.
+
+Not providing full promotion on ContentDB, or not allowing your package at all,
+doesn't mean you can't make such content - it just means we're not going to help
+you spread it.
+
+## What's so bad about licenses that forbid commercial use?
+
+Please read [reasons not to use a Creative Commons -NC license](https://freedomdefined.org/Licenses/NC).
+Here's a quick summary related to Minetest content:
+
+1. They make your work incompatible with a growing body of free content, even if
+   you do want to allow derivative works or combinations.
+   This means that it can cause problems when another modder wishes to include your
+   work in a modpack or game.
+2. They may rule out other basic and beneficial uses that you want to allow.
+   For example, CC -NC will forbid showing your content in a monetised YouTube
+   video.
+3. They are unlikely to increase the potential profit from your work, and a
+   share-alike license serves the goal to protect your work from unethical
+   exploitation equally well.
+
+## How can I show non-free packages in the client?
+
+Non-free packages are hidden in the client by default, partly in order to comply
+with the rules of various Linux distributions.
+
+Users can opt-in to showing non-free software, if they wish:
+
+1. In the main menu, go to Settings > All settings
+2. Search for "ContentDB Flag Blacklist".
+3. Edit that setting to remove `nonfree, `.
+
+<figure class="figure my-4">
+	<img class="figure-img img-fluid rounded" src="/static/contentdb_flag_blacklist.png" alt="Screenshot of the ContentDB Flag Blacklist setting">
+	<figcaption class="figure-caption">Screenshot of the ContentDB Flag Blacklist setting</figcaption>
+</figure>
+
+In the future, [the `platform_default` flag](/help/content_flags/) will be used to control what content
+each platforms shows - Android is significantly stricter about mature content.
+You may wish to remove all text from that setting completely, leaving it blank,
+if you wish to view all content when this happens. Currently, [mature content is
+not permitted on ContentDB](/policy_and_guidance/).

app/flatpages/help/package_config.md

@@ -0,0 +1,125 @@
+title: Package Configuration and Releases Guide
+
+## Introduction
+
+ContentDB will read configuration files in your package when doing several
+tasks, including package and release creation. This page details how you can use
+this to your advantage.
+
+## .conf files
+
+### What is a content .conf file?
+
+Every type of content can have a `.conf` file that contains the metadata.
+
+The filename of the `.conf` file depends on the content type:
+
+* `mod.conf` for mods.
+* `modpack.conf` for mod packs.
+* `game.conf` for games.
+* `texture_pack.conf` for texture packs.
+
+The `.conf` uses a key-value format, separated using equals. Here's a simple example:
+
+	name = mymod
+	description = A short description to show in the client.
+
+### Understood values
+
+ContentDB understands the following information:
+
+* `description` - A short description to show in the client.
+* `depends` - Comma-separated hard dependencies.
+* `optional_depends` - Comma-separated soft dependencies.
+* `min_minetest_version` - The minimum Minetest version this runs on, see [Min and Max Minetest Versions](#min_max_versions).
+* `max_minetest_version` - The maximum Minetest version this runs on, see [Min and Max Minetest Versions](#min_max_versions).
+
+and for mods only:
+
+* `name` - the mod technical name.
+
+
+## .cdb.json
+
+You can include a `.cdb.json` file in the root of your content directory (ie: next to a .conf)
+to update the package meta.
+
+It should be a JSON dictionary with one or more of the following optional keys:
+
+* `type`: One of `GAME`, `MOD`, `TXP`.
+* `title`: Human-readable title.
+* `name`: Technical name (needs permission if already approved).
+* `short_description`
+* `tags`: List of tag names, see [/api/tags/](/api/tags/).
+* `content_warnings`: List of content warning names, see [/api/content_warnings/](/api/content_warnings/).
+* `license`: A license name, see [/api/licenses/](/api/licenses/).
+* `media_license`: A license name.          
+* `long_description`: Long markdown description.
+* `repo`: Git repo URL.
+* `website`: Website URL.
+* `issue_tracker`: Issue tracker URL.
+* `forums`: forum topic ID.
+
+Use `null` to unset fields where relevant.
+
+Example:
+
+```json
+{
+    "title": "Foo bar",
+    "tags": ["pvp", "survival"],
+    "license": "MIT",
+    "website": null
+}
+```
+
+## Controlling Release Creation
+
+### Git-based Releases and Submodules
+
+ContentDB can create releases from a Git repository.
+It will include submodules in the resulting archive.
+Simply set VCS Repository in the package's meta to a Git repository, and then
+choose Git as the method when creating a release.
+
+### Automatic Release Creation
+
+See [Git Update Detection](/help/update_config/).
+You can also use [GitLab/GitHub webhooks](/help/release_webhooks/) or the [API](/help/api/)
+to create releases.
+
+### Min and Max Minetest Versions
+
+<a name="min_max_versions" />
+
+When creating a release, the `.conf` file will be read to determine what Minetest
+versions the release supports. If the `.conf` doesn't specify, then it is assumed
+that it supports all versions.
+
+This happens when you create a release via the ContentDB web interface, the
+[API](/help/api/), or using a [GitLab/GitHub webhook](/help/release_webhooks/).
+
+Here's an example config:
+
+	name = mymod
+	min_minetest_version = 5.0
+	max_minetest_version = 5.3
+
+Leaving out min or max to have them set as "None".
+
+### Excluding files
+
+When using Git to create releases,
+you can exclude files from a release by using [gitattributes](https://git-scm.com/docs/gitattributes):
+
+
+	.*		export-ignore
+	sources		export-ignore
+	*.zip		export-ignore
+
+
+This will prevent any files from being included if they:
+
+* Beginning with `.`
+* or are named `sources` or are inside any directory named `sources`.
+* or have an extension of "zip".

app/flatpages/help/package_tags.md

@@ -1,33 +0,0 @@
-title: Package Tags
-
-## Overview
-
-Tags should be added to packages to enable easy identification of different types of mods, games and texture packs. 
-
-They are only beneficial when applied correctly, so please use the following guidelines.
-
-## Tag Usage
-
-* **Building** - For mods that focus on adding new materials or nodes to build with.
-* **Education** - For mods or games created for educational purposes.
-* **Environment** - For mods that add environmental effects, including ambient sound and weather effects.
-* **Inventory** - For mods that add new inventory systems or new inventory pages.
-* **Machines and Electronics** - For mods that include placeable machinery or electronic components which interact to complete tasks.
-* **Maintenance** - For mods that assist with world or player maintenance. This includes large-scale map manipulation, area protection and other administrative tools.
-* **Mapgen** - For mods that add new biomes, new mapgen decorations, or any other mapgen elements.
-* **Mobs and NPCs** - For mods that add mobs or NPCs, or provide tools that assist with mob and NPC creation or manipulation.
-* **Plants and Farming** - For mods that add new plants or other farmable resources. 
-* **Player effects/Food** - For mods that change player effects, for example speed, jump height or gravity, and food.
-* **Tools** - For mods that add new tools or new features for existing tools.
-* **Transport** - For mods that add transportation methods. This includes teleportation, vehicles and ridable mobs.
-* **Survival** - For mods written specifically for survival games. For example, these mods might focus on game-balance or increase the difficulty level. This tag should also be used for games with a heavy survival focus.
-* **Creative** - For mods written specifically (and often exclusively) for use in creative mode. For example, these mods may add a large amount of decorative content, or content without crafting recipes. This tag should also be used for games with a heavy creative focus.
-* **Multiplayer-focused** - For games that can be played with other players.
-* **Singleplayer-focused** - For games that can be played alone.
-* **PvP** - For games designed to be played competitively against other players.
-* **PvE** - For games designed for one or multiple players which focus on combat against mobs or NPCs.
-* **Puzzle** - For mods and games with a focus on puzzle solving instead of combat.
-* **16px** - For 16px texture packs.
-* **32px** - For 32px texture packs.
-* **64px** - For 64px texture packs.
-* **128px+** - For 128px or higher texture packs.

app/flatpages/help/ranks_permissions.md

@@ -3,24 +3,24 @@ title: Ranks and Permissions
 ## Overview
 
 * **New Members** - mostly untrusted, cannot change package meta data or publish releases without approval.
-* **Members** - Trusted to change the meta data of their own packages', but cannot publish releases.
-* **Trusted Members** - Same as above, but can approve their own releases and packages.
-* **Editors** - Trusted to change the meta data of any package, and also make and publish releases.
+* **Members** - Trusted to change the meta data of their own packages', but cannot approve their own packages.
+* **Trusted Members** - Same as above, but can approve their own releases.
+* **Editors** - Trusted to edit any package or release, and also responsible for approving new packages.
 * **Moderators** - Same as above, but can manage users.
 * **Admins** - Full access.
 
 ## Breakdown
 
-<table class="fancyTable">
+<table class="table table-striped ranks-table">
 	<thead>
 		<tr>
 			<th>Rank</th>
-			<th colspan=2>New Member</th>
-			<th colspan=2>Member</th>
-			<th colspan=2>Trusted Member</th>
-			<th colspan=2>Editor</th>
-			<th colspan=2>Moderator</th>
-			<th colspan=2>Admin</th>
+			<th colspan=2 class="NEW_MEMBER">New Member</th>
+			<th colspan=2 class="MEMBER">Member</th>
+			<th colspan=2 class="TRUSTED_MEMBER">Trusted</th>
+			<th colspan=2 class="EDITOR">Editor</th>
+			<th colspan=2 class="MODERATOR">Moderator</th>
+			<th colspan=2 class="ADMIN">Admin</th>
 		</tr>
 		<tr>
 			<th>Owner of thing</th>
@@ -41,218 +41,232 @@ title: Ranks and Permissions
 	<tbody>
 		<tr>
 			<td>Create Package</td>
-			<th>✓</th> <!-- new -->
-			<th></th>
-			<th>✓</th> <!-- member -->
-			<th></th>
-			<th>✓</th> <!-- trusted member -->
-			<th></th>
-			<th>✓</th> <!-- editor -->
-			<th>✓</th>
-			<th>✓</th> <!-- moderator -->
-			<th>✓</th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
+			<td>✓</td> <!-- new -->
+			<td></td>
+			<td>✓</td> <!-- member -->
+			<td></td>
+			<td>✓</td> <!-- trusted member -->
+			<td></td>
+			<td>✓</td> <!-- editor -->
+			<td>✓</td>
+			<td>✓</td> <!-- moderator -->
+			<td>✓</td>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
 		</tr>
 		<tr>
 			<td>Approve Package</td>
-			<th></th> <!-- new -->
-			<th></th>
-			<th></th> <!-- member -->
-			<th></th>
-			<th></th> <!-- trusted member -->
-			<th></th>
-			<th>✓</th> <!-- editor -->
-			<th>✓</th>
-			<th>✓</th> <!-- moderator -->
-			<th>✓</th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
+			<td></td> <!-- new -->
+			<td></td>
+			<td></td> <!-- member -->
+			<td></td>
+			<td></td> <!-- trusted member -->
+			<td></td>
+			<td>✓</td> <!-- editor -->
+			<td>✓</td>
+			<td>✓</td> <!-- moderator -->
+			<td>✓</td>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
+		</tr>
+		<tr>
+			<td>Delete Package</td>
+			<td></td> <!-- new -->
+			<td></td>
+			<td>✓</td> <!-- member -->
+			<td></td>
+			<td>✓</td> <!-- trusted member -->
+			<td></td>
+			<td>✓</td> <!-- editor -->
+			<td>✓</td>
+			<td>✓</td> <!-- moderator -->
+			<td>✓</td>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
 		</tr>
 		<tr>
 			<td>Edit Package</td>
-			<th></th> <!-- new -->
-			<th></th>
-			<th>✓</th> <!-- member -->
-			<th></th>
-			<th>✓</th> <!-- trusted member -->
-			<th></th>
-			<th>✓</th> <!-- editor -->
-			<th>✓</th>
-			<th>✓</th> <!-- moderator -->
-			<th>✓</th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
+			<td></td> <!-- new -->
+			<td></td>
+			<td>✓</td> <!-- member -->
+			<td></td>
+			<td>✓</td> <!-- trusted member -->
+			<td></td>
+			<td>✓</td> <!-- editor -->
+			<td>✓</td>
+			<td>✓</td> <!-- moderator -->
+			<td>✓</td>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
+		</tr>
+		<tr>
+			<td>Edit Maintainers</td>
+			<td>✓</td> <!-- new -->
+			<td></td>
+			<td>✓</td> <!-- member -->
+			<td></td>
+			<td>✓</td> <!-- trusted member -->
+			<td></td>
+			<td>✓</td> <!-- editor -->
+			<td></td>
+			<td>✓</td> <!-- moderator -->
+			<td>✓</td>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
 		</tr>
 		<tr>
 			<td>Add/Delete Screenshot</td>
-			<th>✓</th> <!-- new -->
-			<th></th>
-			<th>✓</th> <!-- member -->
-			<th></th>
-			<th>✓</th> <!-- trusted member -->
-			<th></th>
-			<th>✓</th> <!-- editor -->
-			<th>✓</th>
-			<th>✓</th> <!-- moderator -->
-			<th>✓</th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
+			<td>✓</td> <!-- new -->
+			<td></td>
+			<td>✓</td> <!-- member -->
+			<td></td>
+			<td>✓</td> <!-- trusted member -->
+			<td></td>
+			<td>✓</td> <!-- editor -->
+			<td>✓</td>
+			<td>✓</td> <!-- moderator -->
+			<td>✓</td>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
 		</tr>
 		<tr>
 			<td>Approve Screenshot</td>
-			<th></th> <!-- new -->
-			<th></th>
-			<th></th> <!-- member -->
-			<th></th>
-			<th>✓</th> <!-- trusted member -->
-			<th></th>
-			<th>✓</th> <!-- editor -->
-			<th>✓</th>
-			<th>✓</th> <!-- moderator -->
-			<th>✓</th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
-		</tr>
-		<tr>
-			<td>Approve EditRequest</td>
-			<th></th> <!-- new -->
-			<th></th>
-			<th>✓</th> <!-- member -->
-			<th></th>
-			<th>✓</th> <!-- trusted member -->
-			<th></th>
-			<th>✓</th> <!-- editor -->
-			<th>✓</th>
-			<th>✓</th> <!-- moderator -->
-			<th>✓</th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
-		</tr>
-		<tr>
-			<td>Edit EditRequest</td>
-			<th>✓<sup>1</sup></th> <!-- new -->
-			<th></th>
-			<th>✓</th> <!-- member -->
-			<th></th>
-			<th>✓</th> <!-- trusted member -->
-			<th></th>
-			<th>✓</th> <!-- editor -->
-			<th>✓</th>
-			<th>✓</th> <!-- moderator -->
-			<th>✓</th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
+			<td></td> <!-- new -->
+			<td></td>
+			<td></td> <!-- member -->
+			<td></td>
+			<td>✓</td> <!-- trusted member -->
+			<td></td>
+			<td>✓</td> <!-- editor -->
+			<td>✓</td>
+			<td>✓</td> <!-- moderator -->
+			<td>✓</td>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
 		</tr>
 		<tr>
 			<td>Make Release</td>
-			<th>✓</th> <!-- new -->
-			<th></th>
-			<th>✓</th> <!-- member -->
-			<th></th>
-			<th>✓</th> <!-- trusted member -->
-			<th></th>
-			<th>✓</th> <!-- editor -->
-			<th>✓</th>
-			<th>✓</th> <!-- moderator -->
-			<th>✓</th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
+			<td>✓</td> <!-- new -->
+			<td></td>
+			<td>✓</td> <!-- member -->
+			<td></td>
+			<td>✓</td> <!-- trusted member -->
+			<td></td>
+			<td>✓</td> <!-- editor -->
+			<td>✓</td>
+			<td>✓</td> <!-- moderator -->
+			<td>✓</td>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
 		</tr>
 		<tr>
 			<td>Approve Release</td>
-			<th></th> <!-- new -->
-			<th></th>
-			<th></th> <!-- member -->
-			<th></th>
-			<th>✓</th> <!-- trusted member -->
-			<th></th>
-			<th>✓</th> <!-- editor -->
-			<th>✓</th>
-			<th>✓</th> <!-- moderator -->
-			<th>✓</th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
+			<td></td> <!-- new -->
+			<td></td>
+			<td>✓</td> <!-- member -->
+			<td></td>
+			<td>✓</td> <!-- trusted member -->
+			<td></td>
+			<td>✓</td> <!-- editor -->
+			<td>✓</td>
+			<td>✓</td> <!-- moderator -->
+			<td>✓</td>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
 		</tr>
 		<tr>
 			<td>Change Release URL</td>
-			<th></th> <!-- new -->
-			<th></th>
-			<th></th> <!-- member -->
-			<th></th>
-			<th></th> <!-- trusted member -->
-			<th></th>
-			<th></th> <!-- editor -->
-			<th></th>
-			<th></th> <!-- moderator -->
-			<th></th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
+			<td></td> <!-- new -->
+			<td></td>
+			<td></td> <!-- member -->
+			<td></td>
+			<td></td> <!-- trusted member -->
+			<td></td>
+			<td></td> <!-- editor -->
+			<td></td>
+			<td></td> <!-- moderator -->
+			<td></td>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
 		</tr>
 		<tr>
 			<td>See Private Thread</td>
-			<th>✓</th> <!-- new -->
-			<th></th>
-			<th>✓</th> <!-- member -->
-			<th></th>
-			<th>✓</th> <!-- trusted member -->
-			<th></th>
-			<th>✓</th> <!-- editor -->
-			<th>✓</th>
-			<th>✓</th> <!-- moderator -->
-			<th>✓</th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
+			<td>✓</td> <!-- new -->
+			<td></td>
+			<td>✓</td> <!-- member -->
+			<td></td>
+			<td>✓</td> <!-- trusted member -->
+			<td></td>
+			<td>✓</td> <!-- editor -->
+			<td>✓</td>
+			<td>✓</td> <!-- moderator -->
+			<td>✓</td>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
+		</tr>
+		<tr>
+			<td>Edit Comments</td>
+			<td></td> <!-- new -->
+			<td></td>
+			<td>✓</td> <!-- member -->
+			<td></td>
+			<td>✓</td> <!-- trusted member -->
+			<td></td>
+			<td>✓</td> <!-- editor -->
+			<td></td>
+			<td>✓</td> <!-- moderator -->
+			<td></td>
+			<td>✓</td> <!-- admin -->
+			<td></td>
 		</tr>
 		<tr>
 			<td>Set Email</td>
-			<th>✓</th> <!-- new -->
-			<th></th>
-			<th>✓</th> <!-- member -->
-			<th></th>
-			<th>✓</th> <!-- trusted member -->
-			<th></th>
-			<th>✓</th> <!-- editor -->
-			<th></th>
-			<th>✓</th> <!-- moderator -->
+			<td>✓</td> <!-- new -->
+			<td></td>
+			<td>✓</td> <!-- member -->
+			<td></td>
+			<td>✓</td> <!-- trusted member -->
+			<td></td>
+			<td>✓</td> <!-- editor -->
+			<td></td>
+			<td>✓</td> <!-- moderator -->
 			<th>✓<sup>2</sup></th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
 		</tr>
 		<tr>
 			<td>Create Token</td>
-			<th></th> <!-- new -->
-			<th></th>
-			<th>✓</th> <!-- member -->
-			<th></th>
-			<th>✓</th> <!-- trusted member -->
-			<th></th>
-			<th>✓</th> <!-- editor -->
-			<th></th>
-			<th>✓</th> <!-- moderator -->
+			<td></td> <!-- new -->
+			<td></td>
+			<td>✓</td> <!-- member -->
+			<td></td>
+			<td>✓</td> <!-- trusted member -->
+			<td></td>
+			<td>✓</td> <!-- editor -->
+			<td></td>
+			<td>✓</td> <!-- moderator -->
 			<th>✓<sup>2</sup></th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
 		</tr>
 		<tr>
 			<td>Set Rank</td>
-			<th></th> <!-- new -->
-			<th></th>
-			<th></th> <!-- member -->
-			<th></th>
-			<th></th> <!-- trusted member -->
-			<th></th>
-			<th></th> <!-- editor -->
-			<th></th>
+			<td></td> <!-- new -->
+			<td></td>
+			<td></td> <!-- member -->
+			<td></td>
+			<td></td> <!-- trusted member -->
+			<td></td>
+			<td></td> <!-- editor -->
+			<td></td>
 			<th>✓<sup>3</sup></th> <!-- moderator -->
 			<th>✓<sup>2</sup><sup>3</sup></th>
-			<th>✓</th> <!-- admin -->
-			<th>✓</th>
+			<td>✓</td> <!-- admin -->
+			<td>✓</td>
 		</tr>
 	</tbody>
 </table>
 
 
-1. User must be the author of the EditRequest.
 2. Target user cannot be an admin.
 3. Cannot set user to a higher rank than themselves.

app/flatpages/help/release_webhooks.md

@@ -9,40 +9,24 @@ ContentDB offers the ability to automatically create releases using webhooks
 from either Github or Gitlab. If you're not using either of those services,
 you can also use the [API](../api) to create releases.
 
+ContentDB also offers the ability to poll a Git repo and check for updates
+without any web hooks, this is limited to once a day.
+See [Git Update Detection](/help/update_config/).
+
 The process is as follows:
 
-1. The user creates an API Token and a webhook to use it. This can be done automatically
-   for Github.
+1. The user creates an API Token and a webhook to use it.
 2. The user pushes a commit to the git host (Gitlab or Github).
 3. The git host posts a webhook notification to ContentDB, using the API token assigned to it.
 4. ContentDB checks the API token and issues a new release.
 
-<p class="alert alert-info">
-	This feature is in beta, and is only available for Trusted Members.
-</p>
-
 ## Setting up
 
-### GitHub (automatic)
+### GitHub
 
-1. Go to your package's page.
-2. Make sure that the repository URL is set to a Github repository.
-   Only github.com is supported.
-3. Go to "Releases" > "+", and click "Setup webhook" at the top of the create release
-   page.
-   If you do not see this, either the repository isn't using Github or you do
-   not have permission to use webhook releases (ie: you're not a Trusted Member).
-4. Grant ContentDB the ability to manage Webhooks.
-5. Set the event to either "New tag or Github Release" (highly recommended) or "Push".
-
-   N.B.: GitHub uses tags to power GitHub Releases, meaning that creating a webhook
-   on "New tag" will sync GitHub and ContentDB releases.
-
-### GitHub (manual)
-
-1. Create an API Token by visiting your profile and clicking "API Tokens: Manage".
+1. Create a ContentDB API Token at [Profile > API Tokens: Manage](/user/tokens/).
 2. Copy the access token that was generated.
-3. Go to the repository's settings > Webhooks > Add Webhook.
+3. Go to the GitLab repository's settings > Webhooks > Add Webhook.
 4. Set the payload URL to `https://content.minetest.net/github/webhook/`
 5. Set the content type to JSON.
 6. Set the secret to the access token that you copied.
@@ -50,32 +34,24 @@ The process is as follows:
   * If you want a rolling release, choose "just the push event".
   * Or if you want a stable release cycle based on tags,
     choose "Let me select" > Branch or tag creation.
+8. Create.
 
-### GitLab (manual)
+### GitLab
 
-1. Create an API Token by visiting your profile and clicking "API Tokens: Manage".
+1. Create a ContentDB API Token at [Profile > API Tokens: Manage](/user/tokens/).
 2. Copy the access token that was generated.
-3. Go to the repository's settings > Integrations.
+3. Go to the GitLab repository's settings > Webhooks.
 4. Set the URL to `https://content.minetest.net/gitlab/webhook/`
-6. Set the secret token to the access token that you copied.
+6. Set the secret token to the ContentDB access token that you copied.
 7. Set the events
     * If you want a rolling release, choose "Push events".
     * Or if you want a stable release cycle based on tags,
       choose "Tag push events".
+8. Add webhook.
 
 ## Configuring
 
-### Setting minimum and maximum Minetest versions
-
-<p class="alert alert-info">
-	This feature is unimplemented.
-</p>
-
-1. Open up the conf file for the package.
-   This will be `game.conf`, `mod.conf`, `modpack.conf`, or `texture_pack.conf`
-   depending on the content type.
-2. Set `min_protocol` and `max_protocol` to the respective protocol numbers
-   of the Minetest versions.
-     * 0.4 = 32
-     * 5.0 = 37
-     * 5.1 = 38
+See the [Package Configuration and Releases Guide](/help/package_config/) for
+documentation on configuring the release creation.
+You can set the min/max Minetest version from the Git repository, and also
+configure what files are included.

app/flatpages/help/top_packages.md

@@ -0,0 +1,36 @@
+title: Top Packages Algorithm
+
+## Package Score
+
+Each package is given a `score`, which is used when ordering them in the
+"Top Games/Mods/Texture Packs" lists. The intention of this feature is
+to make it easier for new users to find good packages.
+
+A package's score is equal to a rolling average of recent downloads,
+plus the sum of the score given by reviews.
+
+A review score is 100 if positive, -100 if negative.
+
+```c
+reviews_sum = sum(100 * (positive ? 1 : -1));
+score       = avg_downloads + reviews_sum;
+```
+
+## Pseudo rolling average of downloads
+
+Each package adds 1 to `avg_downloads` for each unique download,
+and then loses 5% (=1/20) of the value every day.
+
+This is called a [Frecency](https://en.wikipedia.org/wiki/Frecency) heuristic,
+a measure which combines both frequency and recency.
+
+"Unique download" is counted per IP per package.
+Downloading an update won't increase the download count if it's already been
+downloaded from that IP.
+
+## Transparency and Feedback
+
+You can see all scores using the [scores REST API](/api/scores/), or by
+using the [Prometheus metrics](/help/metrics/) endpoint.
+
+Consider [suggesting improvements](https://github.com/minetest/contentdb/issues/new?assignees=&labels=Policy&template=policy.md&title=).

app/flatpages/help/update_config.md

@@ -0,0 +1,43 @@
+title: Git Update Detection
+
+## Introduction
+
+When you push a change to your Git repository, ContentDB can create a new release automatically or
+send you a reminder. ContentDB will check your Git repository one per day, but you can use
+webhooks or the API for faster updates.
+
+Git Update Detection is clever enough to not create a release again if you've already created
+it manually or using webhooks/the API.
+
+## Setting up
+
+* Set "VCS Repository URL" in your package.
+* Open the "Configure Git Update Detection" page:
+    * Go to the Create Release page and click "Set up" on the banner.
+    * If the "How do you want to create releases?" wizard appears, choose "Automatic".
+* Choose a trigger:
+    * **New Commit** - this will trigger for each pushed commit on the default branch, or the branch you specify.
+    * **New Tag** - this will trigger when a New Tag is created.
+* Choose action to occur when the trigger happens:
+    * **Notification** - All maintainers receive a notification under the Bot category, and the package
+      will appear under "Outdated Packages" in [your to do list](/user/todo/).
+    * **Create Release** - A new release is created.
+      If New Commit, the title will be the iso date (eg: 2021-02-01).
+      If New Tag, the title will the tag name.
+
+## Marking a package as up-to-date
+
+Git Update Detection shouldn't erroneously mark packages as outdated if it is configured currently,
+so the first thing you should do is make sure the Update Settings are set correctly.
+
+There are some situations where the settings are correct, but you want to mark a package as
+up-to-date - for example, if you don't want to make a release for a particular tag.
+Clicking "Save" on "Update Settings" will mark a package as up-to-date.
+
+## Configuring Release Creation
+
+See the [Package Configuration and Releases Guide](/help/package_config/) for
+documentation on configuring the release creation.
+
+From the Git repository, you can set the min/max Minetest versions, which files are included,
+and update the package meta.

app/flatpages/help/wtfpl.md

@@ -1,5 +1,5 @@
 title: WTFPL is a terrible license
-no_h1: true
+toc: False
 
 <div id="warning" class="alert alert-warning">
 	<span class="icon_message"></span>
@@ -20,8 +20,6 @@ no_h1: true
 	</script>
 </div>
 
-# WTFPL is a terrible license
-
 The use of WTFPL as a license is discouraged for multiple reasons.
 
 * **No Warranty disclaimer:** This could open you up to being sued.<sup>[1]</sup>

app/flatpages/policy_and_guidance.md

@@ -1,23 +1,17 @@
 title: Package Inclusion Policy and Guidance
 
-<div class="alert alert-warning">
-	<b>Note:</b> This is a draft
-</div>
-
 ## 0. Overview
 
 ContentDB is for the community, and as such listings should be useful to the
 community. To help with this, there are a few rules to improve the quality of
 the listings and to combat abuse.
 
-* No inappropriate content. <sup>2.1</sup>
-* Content must be playable/useful, but not necessarily finished. <sup>2.2</sup>
-* Don't use the name of another mod unless your mod is a fork or reimplementation. <sup>3</sup>
-* Licenses must allow derivatives, redistribution, and must not discriminate. <sup>4</sup>
-* Don't put promotions or advertisements in package listings, except for
-  donation and personal website links which are permitted in the
-  long description. <sup>5</sup>
-* The ContentDB admin reserves the right to remove packages for any reason,
+* **No inappropriate content.** <sup>2.1</sup>
+* **Content must be playable/useful, but not necessarily finished.** <sup>2.2</sup>
+* **Don't use the name of another mod unless your mod is a fork or reimplementation.** <sup>3</sup>
+* **Licenses must allow derivatives, redistribution, and must not discriminate.** <sup>4</sup>
+* **Don't put promotions or advertisements in any package metadata.** <sup>5</sup>
+* **The ContentDB admin reserves the right to remove packages for any reason**,
   including ones not covered by this document, and to ban users who abuse
   this service. <sup>1</sup>
 
@@ -27,27 +21,26 @@ the listings and to combat abuse.
 The ContentDB admin reserves the right to remove packages for any reason,
 including ones not covered by this document, and to ban users who abuse this service.
 
-Also see the [help page on tags](/help/package_tags/).
-
 
 ## 2. Accepted Content
 
 ### 2.1. Acceptable Content
 
 Sexually-orientated content is not permitted.
+If in doubt at what this means, [contact us by raising a report](/help/reporting/).
 
-Mature content, including that relating to drugs, excessive gore, violence, or
-excessive horror, is not currently permitted - but will be in the future.
+Mature content is permitted providing that it is labelled correctly.
+See [Content Flags](/help/content_flags/).
 
-The submission of malware is strictly prohibited. This includes software which
-does not do as it advertises, for example if it posts telemetry without stating
+The submission of malware is strictly prohibited. This includes software that
+does not do as it advertises, for example, if it posts telemetry without stating
 clearly that it does in the package meta.
 
 ### 2.2. State of Completion
 
 ContentDB should only currently contain playable content - content which is
-sufficiently complete to be useful to end users. It's fine to add stuff which
-is still a work in progress (WIP) as long as it adds sufficient value -
+sufficiently complete to be useful to end-users. It's fine to add stuff which
+is still a Work in Progress (WIP) as long as it adds sufficient value;
 MineClone 2 is a good example of a WIP package which may break between releases
 but still has value. Note that this doesn't mean that you should add a thing
 you started working on yesterday, it's worth adding all the basic stuff to
@@ -62,8 +55,10 @@ libraries allows them to be installed when a mod depends on it.
 
 ### 3.1 Right to a name
 
+A package uses a name when it has that name or contains a mod that uses that name.
+
 The first package to use a name based on the creation of its forum topic or
-contentdb submission has the right to the technical name. The use of a package
+ContentDB submission has the right to the technical name. The use of a package
 on a server or in private doesn't reserve its name. No other packages of the same
 type may use the same name, except for the exception given by 3.2.
 
@@ -76,10 +71,10 @@ to change the name of the package, or your package won't be accepted.
 
 We reserve the right to issue exceptions for this where we feel necessary.
 
-### 3.2 Mod Forks and Reimplementations
+### 3.2. Mod Forks and Reimplementations
 
 An exception to the above is that mods are allowed to have the same name as a
-mod if its a fork of that mod (or a close reimplementation). In real terms, it
+mod if it's a fork of that mod (or a close reimplementation). In real terms, it
 should be possible to use the new mod as a drop-in replacement.
 
 We reserve the right to decide whether a mod counts as a fork or
@@ -88,14 +83,14 @@ reimplementation of the mod that owns the name.
 
 ## 4. Licenses
 
-### 4.1 Allowed Licenses
+### 4.1. Allowed Licenses
 
 Please ensure that you correctly credit any resources (code, assets, or otherwise)
 that you have used in your package.
 
-**The use of licenses which do not allow derivatives or redistribution is not
+**The use of licenses that do not allow derivatives or redistribution is not
 permitted. This includes CC-ND (No-Derivatives) and lots of closed source licenses.
-The use of licenses which discriminate between groups of people or forbid the use
+The use of licenses that discriminate between groups of people or forbid the use
 of the content on servers or singleplayer is also not permitted.**
 
 However, closed sourced licenses are allowed if they allow the above.
@@ -106,17 +101,21 @@ get around to adding it.
 Please note that the definitions of "free" and "non-free" is the same as that
 of the [Free Software Foundation](https://www.gnu.org/philosophy/free-sw.en.html).
 
-### 4.2 Recommended Licenses
+### 4.2. Recommended Licenses
 
-It is highly recommended that you use a free and open source software license.
-FOSS licenses result in a sharing community and will increase the number of potential users your package has.
-Using a closed source license will result in your package being massively penalised in the search results and package lists.
+It is highly recommended that you use a Free and Open Source software (FOSS)
+license. FOSS licenses result in a sharing community and will increase the
+number of potential users your package has. Using a closed source license will
+result in your package being massively penalised in the search results and
+package lists. See the help page on [non-free licenses](/help/non_free/) for more
+information.
 
 It is recommended that you use a proper license for code with a warranty
 disclaimer, such as the (L)GPL or MIT. You should also use a proper media license
 for media, such as a Creative Commons license.
 
-The use of WTFPL is discouraged as it doesn't contain a [valid warranty disclaimer](https://cubicspot.blogspot.com/2017/04/wtfpl-is-harmful-to-software-developers.html),
+The use of WTFPL is discouraged as it doesn't contain a
+[valid warranty disclaimer](https://cubicspot.blogspot.com/2017/04/wtfpl-is-harmful-to-software-developers.html),
 and also includes swearing which prevents settings like schools from using your content.
 [Read more](/help/wtfpl/).
 
@@ -125,8 +124,8 @@ Public domain is not a valid license in many countries, please use CC0 or MIT in
 
 ## 5. Promotions and Advertisements (inc. asking for donations)
 
-You may note place any promotions or advertisements in any meta data including
-screensthos. This includes asking for donations, promoting online shops,
+You may not place any promotions or advertisements in any meta data including
+screenshots. This includes asking for donations, promoting online shops,
 or linking to personal websites and social media. Please instead use the
 fields provided on your user profile page to place links to websites and
 donation pages.

app/flatpages/privacy_policy.md

@@ -0,0 +1,93 @@
+title: Privacy Policy
+
+## What Information is Collected
+
+**All users:**
+
+* HTTP requests are logged, with the following information:
+	* Time
+	* IP address
+	* Page URL
+	* Response status code
+
+**With an account:**
+
+* Email address
+* Passwords (hashed and salted using BCrypt)
+* Profile information, such as website URLs and donation URLs
+* Comments and threads
+* Audit log actions (such as edits and logins) and their time stamps
+
+ContentDB collects usernames of content creators from the forums,
+as this is required to index forum topics.
+
+Packages, including releases, screenshots, and any meta information,
+are not considered personal information.
+
+Please avoid giving other personal information as we do not want it.
+
+## How this information is used
+
+* Logged HTTP requests may be used for debugging ContentDB.
+* Email addresses are used to:
+    * Provide essential system messages, such as password resets.
+    * Send notifications - the user may configure this to their needs, including opting out.
+* Passwords are used to authenticate the user.
+* The audit log is used to record actions that may be harmful
+* Other information is displayed as part of ContentDB's service.
+
+## Who has access
+
+* Only the admin has access to the HTTP requests.
+  The logs may be shared with others to aid in debugging, but care will be taken to remove any personal information.
+* Encrypted backups may be shared with selected Minetest staff members (moderators + core devs).
+  The keys and the backups themselves are given to different people,
+  requiring at least two staff members to read a backup.
+* Emails are visible to moderators and the admin.
+  They have access to assist users, and they are not permitted to share email addresses.
+* Hashing protects passwords from being read whilst stored in the database or in backups.
+* Profile information is public, including URLs and linked accounts.
+* The visibility of comments depends on the visibility of threads.
+  They are either public, or visible only to the package author and editors.
+* The complete audit log is visible to moderators.
+  Users may see their own audit log actions on their account settings page.
+  Owners, maintainers, and editors may be able to see the actions on a package in the future.
+* We may be required to share information with law enforcement.
+
+## Location
+
+The ContentDB production server is currently located in Canada.
+Backups are stored in the UK.
+Encrypted backups may be stored in other countries, such as the US or EU.
+
+By using this service, you give permission for the data to be moved as needed.
+
+## Period of Retention
+
+The server uses log rotation, meaning that any logged HTTP requests will be
+forgotten within a few weeks.
+
+Usernames may be kept indefinitely, but other user information will be deleted if
+requested. See below.
+
+## Removal Requests
+
+Please [raise a report](https://content.minetest.net/help/reporting/) if you
+wish to remove your personal information.
+
+ContentDB keeps a record of each username and forum topic on the forums,
+for use in indexing mod/game topics. ContentDB also requires the use of a username
+to uniquely identify a package. Therefore, an author cannot be removed completely
+from ContentDB if they have any packages or mod/game topics on the forum.
+
+If we are unable to remove your account for one of the above reasons, your user
+account will instead be wiped and deactivated, ending up exactly like an author
+who has not yet joined ContentDB. All personal information will be removed from the profile,
+and any comments or threads will be deleted.
+
+## Future Changes to Privacy Policy
+
+We will alert any future changes to the privacy policy via email and
+via notices on the ContentDB website.
+
+By continuing to use this service, you agree to the privacy policy.

app/logic/LogicError.py

@@ -0,0 +1,24 @@
+# ContentDB
+# Copyright (C) 2021 rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+class LogicError(Exception):
+	def __init__(self, code, message):
+		self.code = code
+		self.message = message
+
+	def __str__(self):
+		return repr("LogicError {}: {}".format(self.code, self.message))

app/logic/packages.py

@@ -0,0 +1,169 @@
+# ContentDB
+# Copyright (C) 2021 rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import re
+import validators
+
+from app.logic.LogicError import LogicError
+from app.models import User, Package, PackageType, MetaPackage, Tag, ContentWarning, db, Permission, AuditSeverity, License
+from app.utils import addAuditLog
+
+
+def check(cond: bool, msg: str):
+	if not cond:
+		raise LogicError(400, msg)
+
+
+def get_license(name):
+	if type(name) == License:
+		return name
+
+	license = License.query.filter(License.name.ilike(name)).first()
+	if license is None:
+		raise LogicError(400, "Unknown license: " + name)
+	return license
+
+
+name_re = re.compile("^[a-z0-9_]+$")
+
+any = "?"
+ALLOWED_FIELDS = {
+	"type": any,
+	"title": str,
+	"name": str,
+	"short_description": str,
+	"short_desc": str,
+	"tags": list,
+	"content_warnings": list,
+	"license": any,
+	"media_license": any,
+	"long_description": str,
+	"desc": str,
+	"repo": str,
+	"website": str,
+	"issue_tracker": str,
+	"issueTracker": str,
+	"forums": int,
+}
+
+ALIASES = {
+	"short_description": "short_desc",
+	"issue_tracker": "issueTracker",
+	"long_description": "desc"
+}
+
+
+def is_int(val):
+	try:
+		int(val)
+		return True
+	except ValueError:
+		return False
+
+
+def validate(data: dict):
+	for key, value in data.items():
+		if value is not None:
+			typ = ALLOWED_FIELDS.get(key)
+			check(typ is not None, key + " is not a known field")
+			if typ != any:
+				check(isinstance(value, typ), key + " must be a " + typ.__name__)
+
+	if "name" in data:
+		name = data["name"]
+		check(isinstance(name, str), "Name must be a string")
+		check(bool(name_re.match(name)),
+			"Name can only contain lower case letters (a-z), digits (0-9), and underscores (_)")
+
+	for key in ["repo", "website", "issue_tracker", "issueTracker"]:
+		value = data.get(key)
+		if value is not None:
+			check(value.startswith("http://") or value.startswith("https://"),
+					key + " must start with http:// or https://")
+
+			check(validators.url(value, public=True), key + " must be a valid URL")
+
+
+def do_edit_package(user: User, package: Package, was_new: bool, data: dict, reason: str = None):
+	if not package.checkPerm(user, Permission.EDIT_PACKAGE):
+		raise LogicError(403, "You do not have permission to edit this package")
+
+	if "name" in data and package.name != data["name"] and \
+			not package.checkPerm(user, Permission.CHANGE_NAME):
+		raise LogicError(403, "You do not have permission to change the package name")
+
+	for alias, to in ALIASES.items():
+		if alias in data:
+			data[to] = data[alias]
+
+	validate(data)
+
+	if "type" in data:
+		data["type"] = PackageType.coerce(data["type"])
+
+	if "license" in data:
+		data["license"] = get_license(data["license"])
+
+	if "media_license" in data:
+		data["media_license"] = get_license(data["media_license"])
+
+	for key in ["name", "title", "short_desc", "desc", "type", "license", "media_license",
+			"repo", "website", "issueTracker", "forums"]:
+		if key in data:
+			setattr(package, key, data[key])
+
+	if package.type == PackageType.TXP:
+		package.license = package.media_license
+
+	if was_new and package.type == PackageType.MOD:
+		m = MetaPackage.GetOrCreate(package.name, {})
+		package.provides.append(m)
+
+	if "tags" in data:
+		package.tags.clear()
+		for tag_id in data["tags"]:
+			if is_int(tag_id):
+				package.tags.append(Tag.query.get(tag_id))
+			else:
+				tag = Tag.query.filter_by(name=tag_id).first()
+				if tag is None:
+					raise LogicError(400, "Unknown tag: " + tag_id)
+				package.tags.append(tag)
+
+	if "content_warnings" in data:
+		package.content_warnings.clear()
+		for warning_id in data["content_warnings"]:
+			if is_int(warning_id):
+				package.content_warnings.append(ContentWarning.query.get(warning_id))
+			else:
+				warning = ContentWarning.query.filter_by(name=warning_id).first()
+				if warning is None:
+					raise LogicError(400, "Unknown warning: " + warning_id)
+				package.content_warnings.append(warning)
+
+	if not was_new:
+		if reason is None:
+			msg = "Edited {}".format(package.title)
+		else:
+			msg = "Edited {} ({})".format(package.title, reason)
+
+		severity = AuditSeverity.NORMAL if user in package.maintainers else AuditSeverity.EDITOR
+		addAuditLog(severity, user, msg, package.getDetailsURL(), package)
+
+	db.session.commit()
+
+	return package

app/logic/releases.py

@@ -0,0 +1,90 @@
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import datetime
+
+from celery import uuid
+
+from app.logic.LogicError import LogicError
+from app.logic.uploads import upload_file
+from app.models import PackageRelease, db, Permission, User, Package, MinetestRelease
+from app.tasks.importtasks import makeVCSRelease, checkZipRelease
+from app.utils import AuditSeverity, addAuditLog, nonEmptyOrNone
+
+
+def check_can_create_release(user: User, package: Package):
+	if not package.checkPerm(user, Permission.MAKE_RELEASE):
+		raise LogicError(403, "You do not have permission to make releases")
+
+	five_minutes_ago = datetime.datetime.now() - datetime.timedelta(minutes=5)
+	count = package.releases.filter(PackageRelease.releaseDate > five_minutes_ago).count()
+	if count >= 2:
+		raise LogicError(429, "Too many requests, please wait before trying again")
+
+
+def do_create_vcs_release(user: User, package: Package, title: str, ref: str,
+		min_v: MinetestRelease = None, max_v: MinetestRelease = None, reason: str = None):
+	check_can_create_release(user, package)
+
+	rel = PackageRelease()
+	rel.package = package
+	rel.title   = title
+	rel.url     = ""
+	rel.task_id = uuid()
+	rel.min_rel = min_v
+	rel.max_rel = max_v
+	db.session.add(rel)
+
+	if reason is None:
+		msg = "Created release {}".format(rel.title)
+	else:
+		msg = "Created release {} ({})".format(rel.title, reason)
+	addAuditLog(AuditSeverity.NORMAL, user, msg, package.getDetailsURL(), package)
+
+	db.session.commit()
+
+	makeVCSRelease.apply_async((rel.id, nonEmptyOrNone(ref)), task_id=rel.task_id)
+
+	return rel
+
+
+def do_create_zip_release(user: User, package: Package, title: str, file,
+		min_v: MinetestRelease = None, max_v: MinetestRelease = None, reason: str = None):
+	check_can_create_release(user, package)
+
+	uploaded_url, uploaded_path = upload_file(file, "zip", "a zip file")
+
+	rel = PackageRelease()
+	rel.package = package
+	rel.title   = title
+	rel.url     = uploaded_url
+	rel.task_id = uuid()
+	rel.min_rel = min_v
+	rel.max_rel = max_v
+	db.session.add(rel)
+
+	if reason is None:
+		msg = "Created release {}".format(rel.title)
+	else:
+		msg = "Created release {} ({})".format(rel.title, reason)
+	addAuditLog(AuditSeverity.NORMAL, user, msg, package.getDetailsURL(), package)
+
+	db.session.commit()
+
+	checkZipRelease.apply_async((rel.id, uploaded_path), task_id=rel.task_id)
+
+	return rel

app/logic/screenshots.py

@@ -0,0 +1,58 @@
+import datetime
+
+from app.logic.LogicError import LogicError
+from app.logic.uploads import upload_file
+from app.models import User, Package, PackageScreenshot, Permission, NotificationType, db, AuditSeverity
+from app.utils import addNotification, addAuditLog
+
+
+def do_create_screenshot(user: User, package: Package, title: str, file, reason: str = None):
+	thirty_minutes_ago = datetime.datetime.now() - datetime.timedelta(minutes=30)
+	count = package.screenshots.filter(PackageScreenshot.created_at > thirty_minutes_ago).count()
+	if count >= 20:
+		raise LogicError(429, "Too many requests, please wait before trying again")
+
+	uploaded_url, uploaded_path = upload_file(file, "image", "a PNG or JPG image file")
+
+	counter = 1
+	for screenshot in package.screenshots.all():
+		screenshot.order = counter
+		counter += 1
+
+	ss = PackageScreenshot()
+	ss.package  = package
+	ss.title    = title or "Untitled"
+	ss.url      = uploaded_url
+	ss.approved = package.checkPerm(user, Permission.APPROVE_SCREENSHOT)
+	ss.order    = counter
+	db.session.add(ss)
+
+	if reason is None:
+		msg = "Created screenshot {}".format(ss.title)
+	else:
+		msg = "Created screenshot {} ({})".format(ss.title, reason)
+
+	addNotification(package.maintainers, user, NotificationType.PACKAGE_EDIT, msg, package.getDetailsURL(), package)
+	addAuditLog(AuditSeverity.NORMAL, user, msg, package.getDetailsURL(), package)
+
+	db.session.commit()
+
+	return ss
+
+
+def do_order_screenshots(_user: User, package: Package, order: [any]):
+	lookup = {}
+	for screenshot in package.screenshots.all():
+		lookup[screenshot.id] = screenshot
+
+	counter = 1
+	for id in order:
+		try:
+			lookup[int(id)].order = counter
+			counter += 1
+		except KeyError as e:
+			raise LogicError(400, "Unable to find screenshot with id={}".format(id))
+		except ValueError as e:
+			raise LogicError(400, "Invalid number: {}".format(id))
+
+	db.session.commit()

app/logic/uploads.py

@@ -0,0 +1,61 @@
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import imghdr
+import os
+
+from app.logic.LogicError import LogicError
+from app.models import *
+from app.utils import randomString
+
+
+def get_extension(filename):
+	return filename.rsplit(".", 1)[1].lower() if "." in filename else None
+
+ALLOWED_IMAGES = {"jpeg", "png"}
+def isAllowedImage(data):
+	return imghdr.what(None, data) in ALLOWED_IMAGES
+
+def upload_file(file, fileType, fileTypeDesc):
+	if not file or file is None or file.filename == "":
+		raise LogicError(400, "Expected file")
+
+	assert os.path.isdir(app.config["UPLOAD_DIR"]), "UPLOAD_DIR must exist"
+
+	isImage = False
+	if fileType == "image":
+		allowedExtensions = ["jpg", "jpeg", "png"]
+		isImage = True
+	elif fileType == "zip":
+		allowedExtensions = ["zip"]
+	else:
+		raise Exception("Invalid fileType")
+
+	ext = get_extension(file.filename)
+	if ext is None or not ext in allowedExtensions:
+		raise LogicError(400, "Please upload " + fileTypeDesc)
+
+	if isImage and not isAllowedImage(file.stream.read()):
+		raise LogicError(400, "Uploaded image isn't actually an image")
+
+	file.stream.seek(0)
+
+	filename = randomString(10) + "." + ext
+	filepath = os.path.join(app.config["UPLOAD_DIR"], filename)
+	file.save(filepath)
+
+	return "/uploads/" + filename, filepath

app/maillogger.py

@@ -1,6 +1,7 @@
 import logging
-from enum import Enum
-from app.tasks.emails import sendEmailRaw
+
+from app.tasks.emails import send_user_email
+
 
 def _has_newline(line):
 	"""Used by has_bad_header to check for \\r or \\n"""
@@ -34,76 +35,57 @@ class FlaskMailSubjectFormatter(logging.Formatter):
 class FlaskMailTextFormatter(logging.Formatter):
 	pass
 
-# TODO: hier nog niet tevreden over (vooral logger.error(..., exc_info, stack_info))
 class FlaskMailHTMLFormatter(logging.Formatter):
-	pre_template = "<h1>%s</h1><pre>%s</pre>"
 	def formatException(self, exc_info):
 		formatted_exception = logging.Handler.formatException(self, exc_info)
-		return FlaskMailHTMLFormatter.pre_template % ("Exception information", formatted_exception)
+		return "<pre>%s</pre>" % formatted_exception
 	def formatStack(self, stack_info):
-		return FlaskMailHTMLFormatter.pre_template % ("<h1>Stack information</h1><pre>%s</pre>", stack_info)
+		return "<pre>%s</pre>" % stack_info
 
 
 # see: https://github.com/python/cpython/blob/3.6/Lib/logging/__init__.py (class Handler)
 
 class FlaskMailHandler(logging.Handler):
-	def __init__(self, mailer, subject_template, level=logging.NOTSET):
+	def __init__(self, send_to, subject_template, level=logging.NOTSET):
 		logging.Handler.__init__(self, level)
-		self.mailer = mailer
-		self.send_to = mailer.app.config["MAIL_UTILS_ERROR_SEND_TO"]
+		self.send_to = send_to
 		self.subject_template = subject_template
-		self.html_formatter = None
 
-	def setFormatter(self, text_fmt, html_fmt=None):
+	def setFormatter(self, text_fmt):
 		"""
 		Set the formatters for this handler. Provide at least one formatter.
 		When no text_fmt is provided, no text-part is created for the email body.
 		"""
-		assert (text_fmt, html_fmt) != (None, None), "At least one formatter should be provided"
+		assert text_fmt != None, "At least one formatter should be provided"
 		if type(text_fmt)==str:
 			text_fmt = FlaskMailTextFormatter(text_fmt)
 		self.formatter = text_fmt
-		if type(html_fmt)==str:
-			html_fmt = FlaskMailHTMLFormatter(html_fmt)
-		self.html_formatter = html_fmt
 
 	def getSubject(self, record):
 		fmt = FlaskMailSubjectFormatter(self.subject_template)
 		subject = fmt.format(record)
-		#Since templates can cause header problems, and we rather have a incomplete email then an error, we fix this
+		# Since templates can cause header problems, and we rather have a incomplete email then an error, we fix this
 		if _is_bad_subject(subject):
-			subject="FlaskMailHandler log-entry from %s [original subject is replaced, because it would result in a bad header]" % self.mailer.app.name
+			subject="FlaskMailHandler log-entry from ContentDB [original subject is replaced, because it would result in a bad header]"
 		return subject
 
 	def emit(self, record):
 		text = self.format(record)				if self.formatter	  else None
-		html = self.html_formatter.format(record) if self.html_formatter else None
-		sendEmailRaw.delay(self.send_to, self.getSubject(record), text, html)
+		html = "<pre>{}</pre>".format(text)
+		for email in self.send_to:
+			send_user_email.delay(email, self.getSubject(record), text, html)
 
 
-def register_mail_error_handler(app, mailer):
-	subject_template = "ContentDB crashed (%(module)s > %(funcName)s)"
-	text_template = """
-Message type: %(levelname)s
-Location:	 %(pathname)s:%(lineno)d
-Module:	   %(module)s
-Function:	 %(funcName)s
-Time:		 %(asctime)s
-Message:
-%(message)s"""
-	html_template = """
-<style>th { text-align: right}</style><table>
-<tr><th>Message type:</th><td>%(levelname)s</td></tr>
-<tr>	<th>Location:</th><td>%(pathname)s:%(lineno)d</td></tr>
-<tr>	  <th>Module:</th><td>%(module)s</td></tr>
-<tr>	<th>Function:</th><td>%(funcName)s</td></tr>
-<tr>		<th>Time:</th><td>%(asctime)s</td></tr>
-</table>
-<h2>Message</h2>
-<pre>%(message)s</pre>"""
+def build_handler(app):
+	subject_template = "ContentDB %(message)s (%(module)s > %(funcName)s)"
+	text_template = ("Message type: %(levelname)s\n"
+		"Location: %(pathname)s:%(lineno)d\n"
+		"Module:   %(module)s\n"
+		"Function: %(funcName)s\n"
+		"Time:     %(asctime)s\n"
+		"Message: %(message)s\n\n")
 
-	import logging
-	mail_handler = FlaskMailHandler(mailer, subject_template)
+	mail_handler = FlaskMailHandler(app.config["MAIL_UTILS_ERROR_SEND_TO"], subject_template)
 	mail_handler.setLevel(logging.ERROR)
-	mail_handler.setFormatter(text_template, html_template)
-	app.logger.addHandler(mail_handler)
+	mail_handler.setFormatter(text_template)
+	return mail_handler

app/markdown.py

@@ -1,63 +0,0 @@
-import bleach
-from markdown import Markdown
-from flask import Markup
-
-# Whitelist source: MIT
-#
-# https://github.com/Wenzil/mdx_bleach/blob/master/mdx_bleach/whitelist.py
-
-"""
-Default whitelist of allowed HTML tags. Any other HTML tags will be escaped or
-stripped from the text. This applies to the html output that Markdown produces.
-"""
-ALLOWED_TAGS = [
-	'ul',
-	'ol',
-	'li',
-	'p',
-	'pre',
-	'code',
-	'blockquote',
-	'h1',
-	'h2',
-	'h3',
-	'h4',
-	'h5',
-	'h6',
-	'hr',
-	'br',
-	'strong',
-	'em',
-	'a',
-	'img'
-]
-
-"""
-Default whitelist of attributes. It allows the href and title attributes for <a>
-tags and the src, title and alt attributes for <img> tags. Any other attribute
-will be stripped from its tag.
-"""
-ALLOWED_ATTRIBUTES = {
-	'a': ['href', 'title'],
-	'img': ['src', 'title', 'alt']
-}
-
-"""
-If you allow tags that have attributes containing a URI value
-(like the href attribute of an anchor tag,) you may want to adapt
-the accepted protocols. The default list only allows http, https and mailto.
-"""
-ALLOWED_PROTOCOLS = ['http', 'https', 'mailto']
-
-
-md = Markdown(extensions=["fenced_code"], output_format="html5")
-
-def render_markdown(source):
-	return bleach.clean(md.convert(source), \
-		tags=ALLOWED_TAGS, attributes=ALLOWED_ATTRIBUTES, \
-		styles=[], protocols=ALLOWED_PROTOCOLS)
-
-def init_app(app):
-	@app.template_filter()
-	def markdown(source):
-		return Markup(render_markdown(source))

app/models/__init__.py

@@ -0,0 +1,177 @@
+# ContentDB
+# Copyright (C) 2018-21  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask_migrate import Migrate
+from flask_sqlalchemy import SQLAlchemy
+from sqlalchemy_searchable import make_searchable
+
+from app import app
+
+# Initialise database
+
+db = SQLAlchemy(app)
+migrate = Migrate(app, db)
+make_searchable(db.metadata)
+
+
+from .packages import *
+from .users import *
+from .threads import *
+
+
+class APIToken(db.Model):
+	id           = db.Column(db.Integer, primary_key=True)
+	access_token = db.Column(db.String(34), unique=True, nullable=False)
+
+	name         = db.Column(db.String(100), nullable=False)
+
+	owner_id     = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False)
+	owner        = db.relationship("User", foreign_keys=[owner_id], back_populates="tokens")
+
+	created_at   = db.Column(db.DateTime, nullable=False, default=datetime.datetime.utcnow)
+
+	package_id = db.Column(db.Integer, db.ForeignKey("package.id"), nullable=True)
+	package    = db.relationship("Package", foreign_keys=[package_id], back_populates="tokens")
+
+	def canOperateOnPackage(self, package):
+		if self.package and self.package != package:
+			return False
+
+		return package.author == self.owner
+
+
+class AuditSeverity(enum.Enum):
+	NORMAL = 0 # Normal user changes
+	USER   = 1 # Security user changes
+	EDITOR = 2 # Editor changes
+	MODERATION = 3 # Destructive / moderator changes
+
+	def __str__(self):
+		return self.name
+
+	def getTitle(self):
+		return self.name.replace("_", " ").title()
+
+	@classmethod
+	def choices(cls):
+		return [(choice, choice.getTitle()) for choice in cls]
+
+	@classmethod
+	def coerce(cls, item):
+		return item if type(item) == AuditSeverity else AuditSeverity[item]
+
+
+class AuditLogEntry(db.Model):
+	id         = db.Column(db.Integer, primary_key=True)
+
+	created_at = db.Column(db.DateTime, nullable=False, default=datetime.datetime.utcnow)
+
+	causer_id  = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=True)
+	causer     = db.relationship("User", foreign_keys=[causer_id], back_populates="audit_log_entries")
+
+	severity   = db.Column(db.Enum(AuditSeverity), nullable=False)
+
+	title      = db.Column(db.String(100), nullable=False)
+	url        = db.Column(db.String(200), nullable=True)
+
+	package_id = db.Column(db.Integer, db.ForeignKey("package.id"), nullable=True)
+	package    = db.relationship("Package", foreign_keys=[package_id], back_populates="audit_log_entries")
+
+	description = db.Column(db.Text, nullable=True, default=None)
+
+	def __init__(self, causer, severity, title, url, package=None, description=None):
+		if len(title) > 100:
+			title = title[:99] + "…"
+
+		self.causer   = causer
+		self.severity = severity
+		self.title    = title
+		self.url      = url
+		self.package  = package
+		self.description = description
+
+
+REPO_BLACKLIST = [".zip", "mediafire.com", "dropbox.com", "weebly.com",
+	"minetest.net", "dropboxusercontent.com", "4shared.com",
+	"digitalaudioconcepts.com", "hg.intevation.org", "www.wtfpl.net",
+	"imageshack.com", "imgur.com"]
+
+
+class ForumTopic(db.Model):
+	topic_id  = db.Column(db.Integer, primary_key=True, autoincrement=False)
+
+	author_id = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False)
+	author    = db.relationship("User")
+
+	wip       = db.Column(db.Boolean, server_default="0")
+	discarded = db.Column(db.Boolean, server_default="0")
+
+	type      = db.Column(db.Enum(PackageType), nullable=False)
+	title     = db.Column(db.String(200), nullable=False)
+	name      = db.Column(db.String(30), nullable=True)
+	link      = db.Column(db.String(200), nullable=True)
+
+	posts     = db.Column(db.Integer, nullable=False)
+	views     = db.Column(db.Integer, nullable=False)
+
+	created_at = db.Column(db.DateTime, nullable=False, default=datetime.datetime.utcnow)
+
+	def getRepoURL(self):
+		if self.link is None:
+			return None
+
+		for item in REPO_BLACKLIST:
+			if item in self.link:
+				return None
+
+		return self.link.replace("repo.or.cz/w/", "repo.or.cz/")
+
+	def getAsDictionary(self):
+		return {
+			"author": self.author.username,
+			"name":   self.name,
+			"type":   self.type.toName(),
+			"title":  self.title,
+			"id":     self.topic_id,
+			"link":   self.link,
+			"posts":  self.posts,
+			"views":  self.views,
+			"is_wip": self.wip,
+			"discarded":  self.discarded,
+			"created_at": self.created_at.isoformat(),
+		}
+
+	def checkPerm(self, user, perm):
+		if not user.is_authenticated:
+			return False
+
+		if type(perm) == str:
+			perm = Permission[perm]
+		elif type(perm) != Permission:
+			raise Exception("Unknown permission given to ForumTopic.checkPerm()")
+
+		if perm == Permission.TOPIC_DISCARD:
+			return self.author == user or user.rank.atLeast(UserRank.EDITOR)
+
+		else:
+			raise Exception("Permission {} is not related to topics".format(perm.name))
+
+
+if app.config.get("LOG_SQL"):
+	import logging
+	logging.basicConfig()
+	logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)

app/models/threads.py

@@ -0,0 +1,154 @@
+# ContentDB
+# Copyright (C) 2018-21  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+import datetime
+
+from flask import url_for
+
+from . import db
+from .users import Permission, UserRank
+from .packages import Package
+
+
+watchers = db.Table("watchers",
+	db.Column("user_id", db.Integer, db.ForeignKey("user.id"), primary_key=True),
+	db.Column("thread_id", db.Integer, db.ForeignKey("thread.id"), primary_key=True)
+)
+
+
+class Thread(db.Model):
+	id         = db.Column(db.Integer, primary_key=True)
+
+	package_id = db.Column(db.Integer, db.ForeignKey("package.id"), nullable=True)
+	package    = db.relationship("Package", foreign_keys=[package_id], back_populates="threads")
+
+	is_review_thread = db.relationship("Package", foreign_keys=[Package.review_thread_id], back_populates="review_thread")
+
+	review_id  = db.Column(db.Integer, db.ForeignKey("package_review.id"), nullable=True)
+	review     = db.relationship("PackageReview", foreign_keys=[review_id], cascade="all, delete")
+
+	author_id  = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False)
+	author     = db.relationship("User", back_populates="threads", foreign_keys=[author_id])
+
+	title      = db.Column(db.String(100), nullable=False)
+	private    = db.Column(db.Boolean, server_default="0", nullable=False)
+
+	locked     = db.Column(db.Boolean, server_default="0", nullable=False)
+
+	created_at = db.Column(db.DateTime, nullable=False, default=datetime.datetime.utcnow)
+
+	replies    = db.relationship("ThreadReply", back_populates="thread", lazy="dynamic",
+			order_by=db.asc("thread_reply_id"), cascade="all, delete, delete-orphan")
+
+	watchers   = db.relationship("User", secondary=watchers, backref="watching")
+
+	def getViewURL(self):
+		return url_for("threads.view", id=self.id, _external=False)
+
+	def getSubscribeURL(self):
+		return url_for("threads.subscribe", id=self.id)
+
+	def getUnsubscribeURL(self):
+		return url_for("threads.unsubscribe", id=self.id)
+
+	def checkPerm(self, user, perm):
+		if not user.is_authenticated:
+			return perm == Permission.SEE_THREAD and not self.private
+
+		if type(perm) == str:
+			perm = Permission[perm]
+		elif type(perm) != Permission:
+			raise Exception("Unknown permission given to Thread.checkPerm()")
+
+		isMaintainer = user == self.author or (self.package is not None and self.package.author == user)
+		if self.package:
+			isMaintainer = isMaintainer or user in self.package.maintainers
+
+		canSee = not self.private or isMaintainer or user.rank.atLeast(UserRank.EDITOR)
+
+		if perm == Permission.SEE_THREAD:
+			return canSee
+
+		elif perm == Permission.COMMENT_THREAD:
+			return canSee and (not self.locked or user.rank.atLeast(UserRank.MODERATOR))
+
+		elif perm == Permission.LOCK_THREAD or perm == Permission.DELETE_THREAD:
+			return user.rank.atLeast(UserRank.MODERATOR)
+
+		else:
+			raise Exception("Permission {} is not related to threads".format(perm.name))
+
+	def get_latest_reply(self):
+		return ThreadReply.query.filter_by(thread_id=self.id).order_by(db.desc(ThreadReply.id)).first()
+
+
+class ThreadReply(db.Model):
+	id         = db.Column(db.Integer, primary_key=True)
+
+	thread_id  = db.Column(db.Integer, db.ForeignKey("thread.id"), nullable=False)
+	thread = db.relationship("Thread", back_populates="replies", foreign_keys=[thread_id])
+
+	comment    = db.Column(db.String(2000), nullable=False)
+
+	author_id  = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False)
+	author     = db.relationship("User", back_populates="replies", foreign_keys=[author_id])
+
+	created_at = db.Column(db.DateTime, nullable=False, default=datetime.datetime.utcnow)
+
+	def checkPerm(self, user, perm):
+		if not user.is_authenticated:
+			return False
+
+		if type(perm) == str:
+			perm = Permission[perm]
+		elif type(perm) != Permission:
+			raise Exception("Unknown permission given to ThreadReply.checkPerm()")
+
+		if perm == Permission.EDIT_REPLY:
+			return user == self.author and user.rank.atLeast(UserRank.MEMBER) and not self.thread.locked
+
+		elif perm == Permission.DELETE_REPLY:
+			return user.rank.atLeast(UserRank.MODERATOR) and self.thread.replies[0] != self
+
+		else:
+			raise Exception("Permission {} is not related to threads".format(perm.name))
+
+
+class PackageReview(db.Model):
+	id         = db.Column(db.Integer, primary_key=True)
+
+	package_id = db.Column(db.Integer, db.ForeignKey("package.id"), nullable=True)
+	package    = db.relationship("Package", foreign_keys=[package_id], back_populates="reviews")
+
+	created_at = db.Column(db.DateTime, nullable=False, default=datetime.datetime.utcnow)
+
+	author_id  = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False)
+	author     = db.relationship("User", foreign_keys=[author_id], back_populates="reviews")
+
+	recommends = db.Column(db.Boolean, nullable=False)
+
+	thread     = db.relationship("Thread", uselist=False, back_populates="review")
+
+	def asSign(self):
+		return 1 if self.recommends else -1
+
+	def getEditURL(self):
+		return self.package.getReviewURL()
+
+	def getDeleteURL(self):
+		return url_for("packages.delete_review",
+				author=self.package.author.username,
+				name=self.package.name)

app/models/users.py

@@ -0,0 +1,468 @@
+# ContentDB
+# Copyright (C) 2018-21  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+import datetime
+import enum
+
+from flask_login import UserMixin
+from sqlalchemy import desc, text
+
+from app import gravatar
+from . import db
+
+
+class UserRank(enum.Enum):
+	BANNED         = 0
+	NOT_JOINED     = 1
+	NEW_MEMBER     = 2
+	MEMBER         = 3
+	TRUSTED_MEMBER = 4
+	EDITOR         = 5
+	BOT            = 6
+	MODERATOR      = 7
+	ADMIN          = 8
+
+	def atLeast(self, min):
+		return self.value >= min.value
+
+	def getTitle(self):
+		return self.name.replace("_", " ").title()
+
+	def toName(self):
+		return self.name.lower()
+
+	def __str__(self):
+		return self.name
+
+	@classmethod
+	def choices(cls):
+		return [(choice, choice.getTitle()) for choice in cls]
+
+	@classmethod
+	def coerce(cls, item):
+		return item if type(item) == UserRank else UserRank[item]
+
+
+class Permission(enum.Enum):
+	EDIT_PACKAGE       = "EDIT_PACKAGE"
+	APPROVE_CHANGES    = "APPROVE_CHANGES"
+	DELETE_PACKAGE     = "DELETE_PACKAGE"
+	CHANGE_AUTHOR      = "CHANGE_AUTHOR"
+	CHANGE_NAME        = "CHANGE_NAME"
+	MAKE_RELEASE       = "MAKE_RELEASE"
+	DELETE_RELEASE     = "DELETE_RELEASE"
+	ADD_SCREENSHOTS    = "ADD_SCREENSHOTS"
+	REIMPORT_META      = "REIMPORT_META"
+	APPROVE_SCREENSHOT = "APPROVE_SCREENSHOT"
+	APPROVE_RELEASE    = "APPROVE_RELEASE"
+	APPROVE_NEW        = "APPROVE_NEW"
+	EDIT_TAGS          = "EDIT_TAGS"
+	CREATE_TAG         = "CREATE_TAG"
+	CHANGE_RELEASE_URL = "CHANGE_RELEASE_URL"
+	CHANGE_USERNAMES   = "CHANGE_USERNAMES"
+	CHANGE_RANK        = "CHANGE_RANK"
+	CHANGE_EMAIL       = "CHANGE_EMAIL"
+	SEE_THREAD         = "SEE_THREAD"
+	CREATE_THREAD      = "CREATE_THREAD"
+	COMMENT_THREAD     = "COMMENT_THREAD"
+	LOCK_THREAD        = "LOCK_THREAD"
+	DELETE_THREAD      = "DELETE_THREAD"
+	DELETE_REPLY       = "DELETE_REPLY"
+	EDIT_REPLY         = "EDIT_REPLY"
+	UNAPPROVE_PACKAGE  = "UNAPPROVE_PACKAGE"
+	TOPIC_DISCARD      = "TOPIC_DISCARD"
+	CREATE_TOKEN       = "CREATE_TOKEN"
+	EDIT_MAINTAINERS   = "EDIT_MAINTAINERS"
+	CHANGE_PROFILE_URLS = "CHANGE_PROFILE_URLS"
+
+	# Only return true if the permission is valid for *all* contexts
+	# See Package.checkPerm for package-specific contexts
+	def check(self, user):
+		if not user.is_authenticated:
+			return False
+
+		if self == Permission.APPROVE_NEW or \
+				self == Permission.APPROVE_CHANGES    or \
+				self == Permission.APPROVE_RELEASE    or \
+				self == Permission.APPROVE_SCREENSHOT or \
+				self == Permission.EDIT_TAGS or \
+				self == Permission.CREATE_TAG or \
+				self == Permission.SEE_THREAD:
+			return user.rank.atLeast(UserRank.EDITOR)
+		else:
+			raise Exception("Non-global permission checked globally. Use Package.checkPerm or User.checkPerm instead.")
+
+	@staticmethod
+	def checkPerm(user, perm):
+		if type(perm) == str:
+			perm = Permission[perm]
+		elif type(perm) != Permission:
+			raise Exception("Unknown permission given to Permission.check")
+
+		return perm.check(user)
+
+
+def display_name_default(context):
+	return context.get_current_parameters()["username"]
+
+
+class User(db.Model, UserMixin):
+	id           = db.Column(db.Integer, primary_key=True)
+
+	# User authentication information
+	username     = db.Column(db.String(50, collation="NOCASE"), nullable=False, unique=True, index=True)
+	password     = db.Column(db.String(255), nullable=True, server_default=None)
+	reset_password_token = db.Column(db.String(100), nullable=False, server_default="")
+
+	def get_id(self):
+		return self.username
+
+	rank         = db.Column(db.Enum(UserRank), nullable=False)
+
+	# Account linking
+	github_username = db.Column(db.String(50, collation="NOCASE"), nullable=True, unique=True)
+	forums_username = db.Column(db.String(50, collation="NOCASE"), nullable=True, unique=True)
+
+	# Access token for webhook setup
+	github_access_token = db.Column(db.String(50), nullable=True, server_default=None)
+
+	# User email information
+	email         = db.Column(db.String(255), nullable=True, unique=True)
+	email_confirmed_at  = db.Column(db.DateTime(), nullable=True)
+
+	# User information
+	profile_pic   = db.Column(db.String(255), nullable=True, server_default=None)
+	is_active     = db.Column("is_active", db.Boolean, nullable=False, server_default="0")
+	display_name  = db.Column(db.String(100), nullable=False, default=display_name_default)
+
+	# Links
+	website_url   = db.Column(db.String(255), nullable=True, default=None)
+	donate_url    = db.Column(db.String(255), nullable=True, default=None)
+
+	# Content
+	notifications = db.relationship("Notification", foreign_keys="Notification.user_id",
+			order_by=desc(text("Notification.created_at")), back_populates="user", cascade="all, delete, delete-orphan")
+	caused_notifications = db.relationship("Notification", foreign_keys="Notification.causer_id",
+			back_populates="causer", cascade="all, delete, delete-orphan", lazy="dynamic")
+	notification_preferences = db.relationship("UserNotificationPreferences", uselist=False, back_populates="user",
+			cascade="all, delete, delete-orphan")
+
+	email_verifications = db.relationship("UserEmailVerification", foreign_keys="UserEmailVerification.user_id",
+			back_populates="user", cascade="all, delete, delete-orphan", lazy="dynamic")
+
+	audit_log_entries = db.relationship("AuditLogEntry", foreign_keys="AuditLogEntry.causer_id", back_populates="causer",
+			order_by=desc("audit_log_entry_created_at"), lazy="dynamic")
+
+	maintained_packages = db.relationship("Package", lazy="dynamic", secondary="maintainers", order_by=db.asc("package_title"))
+
+	packages      = db.relationship("Package", back_populates="author", lazy="dynamic", order_by=db.asc("package_title"))
+	reviews       = db.relationship("PackageReview", back_populates="author", order_by=db.desc("package_review_created_at"), cascade="all, delete, delete-orphan")
+	tokens        = db.relationship("APIToken", back_populates="owner", lazy="dynamic", cascade="all, delete, delete-orphan")
+	threads       = db.relationship("Thread", back_populates="author", lazy="dynamic", cascade="all, delete, delete-orphan")
+	replies       = db.relationship("ThreadReply", back_populates="author", lazy="dynamic", cascade="all, delete, delete-orphan")
+
+	def __init__(self, username=None, active=False, email=None, password=None):
+		self.username = username
+		self.email_confirmed_at = datetime.datetime.now() - datetime.timedelta(days=6000)
+		self.display_name = username
+		self.is_active = active
+		self.email = email
+		self.password = password
+		self.rank = UserRank.NOT_JOINED
+
+	def canAccessTodoList(self):
+		return Permission.APPROVE_NEW.check(self) or \
+				Permission.APPROVE_RELEASE.check(self) or \
+				Permission.APPROVE_CHANGES.check(self)
+
+	def isClaimed(self):
+		return self.rank.atLeast(UserRank.NEW_MEMBER)
+
+	def getProfilePicURL(self):
+		if self.profile_pic:
+			return self.profile_pic
+		elif self.rank == UserRank.BOT:
+			return "/static/bot_avatar.png"
+		else:
+			return gravatar(self.email or "")
+
+	def checkPerm(self, user, perm):
+		if not user.is_authenticated:
+			return False
+
+		if type(perm) == str:
+			perm = Permission[perm]
+		elif type(perm) != Permission:
+			raise Exception("Unknown permission given to User.checkPerm()")
+
+		# Members can edit their own packages, and editors can edit any packages
+		if perm == Permission.CHANGE_AUTHOR:
+			return user.rank.atLeast(UserRank.EDITOR)
+		elif perm == Permission.CHANGE_RANK or perm == Permission.CHANGE_USERNAMES:
+			return user.rank.atLeast(UserRank.MODERATOR)
+		elif perm == Permission.CHANGE_EMAIL or perm == Permission.CHANGE_PROFILE_URLS:
+			return user == self or user.rank.atLeast(UserRank.ADMIN)
+		elif perm == Permission.CREATE_TOKEN:
+			if user == self:
+				return user.rank.atLeast(UserRank.MEMBER)
+			else:
+				return user.rank.atLeast(UserRank.MODERATOR) and user.rank.atLeast(self.rank)
+		else:
+			raise Exception("Permission {} is not related to users".format(perm.name))
+
+	def canCommentRL(self):
+		from app.models import ThreadReply
+
+		factor = 1
+		if self.rank.atLeast(UserRank.ADMIN):
+			return True
+		elif self.rank.atLeast(UserRank.TRUSTED_MEMBER):
+			factor *= 2
+
+		one_min_ago = datetime.datetime.utcnow() - datetime.timedelta(minutes=1)
+		if ThreadReply.query.filter_by(author=self) \
+				.filter(ThreadReply.created_at > one_min_ago).count() >= 3 * factor:
+			return False
+
+		hour_ago = datetime.datetime.utcnow() - datetime.timedelta(hours=1)
+		if ThreadReply.query.filter_by(author=self) \
+				.filter(ThreadReply.created_at > hour_ago).count() >= 20 * factor:
+			return False
+
+		return True
+
+	def canOpenThreadRL(self):
+		from app.models import Thread
+
+		factor = 1
+		if self.rank.atLeast(UserRank.ADMIN):
+			return True
+		elif self.rank.atLeast(UserRank.TRUSTED_MEMBER):
+			factor *= 5
+
+		hour_ago = datetime.datetime.utcnow() - datetime.timedelta(hours=1)
+		return Thread.query.filter_by(author=self) \
+			.filter(Thread.created_at > hour_ago).count() < 2 * factor
+
+	def __eq__(self, other):
+		if other is None:
+			return False
+
+		if not self.is_authenticated or not other.is_authenticated:
+			return False
+
+		assert self.id > 0
+		return self.id == other.id
+
+	def can_see_edit_profile(self, current_user):
+		return self.checkPerm(current_user, Permission.CHANGE_USERNAMES) or \
+			self.checkPerm(current_user, Permission.CHANGE_EMAIL) or \
+			self.checkPerm(current_user, Permission.CHANGE_RANK)
+
+	def can_delete(self):
+		from app.models import ForumTopic
+		return self.packages.count() == 0 and ForumTopic.query.filter_by(author=self).count() == 0
+
+
+class UserEmailVerification(db.Model):
+	id      = db.Column(db.Integer, primary_key=True)
+	user_id = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False)
+	email   = db.Column(db.String(100), nullable=False)
+	token   = db.Column(db.String(32), nullable=True)
+	user    = db.relationship("User", foreign_keys=[user_id], back_populates="email_verifications")
+	is_password_reset = db.Column(db.Boolean, nullable=False, default=False)
+
+
+class EmailSubscription(db.Model):
+	id          = db.Column(db.Integer, primary_key=True)
+	email       = db.Column(db.String(100), nullable=False, unique=True)
+	blacklisted = db.Column(db.Boolean, nullable=False, default=False)
+	token       = db.Column(db.String(32), nullable=True, default=None)
+
+	def __init__(self, email):
+		self.email = email
+		self.blacklisted = False
+		self.token = None
+
+
+class NotificationType(enum.Enum):
+	# Package / release / etc
+	PACKAGE_EDIT   = 1
+
+	# Approval review actions
+	PACKAGE_APPROVAL = 2
+
+	# New thread
+	NEW_THREAD     = 3
+
+	# New Review
+	NEW_REVIEW     = 4
+
+	# Posted reply to subscribed thread
+	THREAD_REPLY   = 5
+
+	# A bot notification
+	BOT            = 6
+
+	# Added / removed as maintainer
+	MAINTAINER     = 7
+
+	# Editor misc
+	EDITOR_ALERT   = 8
+
+	# Editor misc
+	EDITOR_MISC    = 9
+
+	# Any other
+	OTHER          = 0
+
+
+	def getTitle(self):
+		return self.name.replace("_", " ").title()
+
+	def toName(self):
+		return self.name.lower()
+
+	def get_description(self):
+		if self == NotificationType.PACKAGE_EDIT:
+			return "When another user edits your packages, releases, etc."
+		elif self == NotificationType.PACKAGE_APPROVAL:
+			return "Notifications from editors related to the package approval process."
+		elif self == NotificationType.NEW_THREAD:
+			return "When a thread is created on your package."
+		elif self == NotificationType.NEW_REVIEW:
+			return "When a user posts a review on your package."
+		elif self == NotificationType.THREAD_REPLY:
+			return "When someone replies to a thread you're watching."
+		elif self == NotificationType.BOT:
+			return "From a bot - for example, update notifications."
+		elif self == NotificationType.MAINTAINER:
+			return "When your package's maintainers change."
+		elif self == NotificationType.EDITOR_ALERT:
+			return "For editors: Important alerts."
+		elif self == NotificationType.EDITOR_MISC:
+			return "For editors: Minor notifications, including new threads."
+		elif self == NotificationType.OTHER:
+			return "Minor notifications not important enough for a dedicated category."
+		else:
+			return ""
+
+	def __str__(self):
+		return self.name
+
+	def __lt__(self, other):
+		return self.value < other.value
+
+	@classmethod
+	def choices(cls):
+		return [(choice, choice.getTitle()) for choice in cls]
+
+	@classmethod
+	def coerce(cls, item):
+		return item if type(item) == NotificationType else NotificationType[item]
+
+
+class Notification(db.Model):
+	id         = db.Column(db.Integer, primary_key=True)
+
+	user_id    = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False)
+	user       = db.relationship("User", foreign_keys=[user_id], back_populates="notifications")
+
+	causer_id  = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False)
+	causer     = db.relationship("User", foreign_keys=[causer_id], back_populates="caused_notifications")
+
+	type       = db.Column(db.Enum(NotificationType), nullable=False, default=NotificationType.OTHER)
+
+	emailed    = db.Column(db.Boolean(), nullable=False, default=False)
+
+	title      = db.Column(db.String(100), nullable=False)
+	url        = db.Column(db.String(200), nullable=True)
+
+	package_id = db.Column(db.Integer, db.ForeignKey("package.id"), nullable=True)
+	package    = db.relationship("Package", foreign_keys=[package_id], back_populates="notifications")
+
+	created_at = db.Column(db.DateTime, nullable=False, default=datetime.datetime.utcnow)
+
+	def __init__(self, user, causer, type, title, url, package=None):
+		if len(title) > 100:
+			title = title[:99] + "…"
+
+		self.user    = user
+		self.causer  = causer
+		self.type    = type
+		self.title   = title
+		self.url     = url
+		self.package = package
+
+	def can_send_email(self):
+		prefs = self.user.notification_preferences
+		return prefs and self.user.email and prefs.get_can_email(self.type)
+
+	def can_send_digest(self):
+		prefs = self.user.notification_preferences
+		return prefs and self.user.email and prefs.get_can_digest(self.type)
+
+
+class UserNotificationPreferences(db.Model):
+	id = db.Column(db.Integer, primary_key=True)
+	user_id = db.Column(db.Integer, db.ForeignKey('user.id'), nullable=False)
+	user = db.relationship("User", back_populates="notification_preferences")
+
+	# 2 = immediate emails
+	# 1 = daily digest emails
+	# 0 = no emails
+
+	pref_package_edit     = db.Column(db.Integer, nullable=False)
+	pref_package_approval = db.Column(db.Integer, nullable=False)
+	pref_new_thread       = db.Column(db.Integer, nullable=False)
+	pref_new_review       = db.Column(db.Integer, nullable=False)
+	pref_thread_reply     = db.Column(db.Integer, nullable=False)
+	pref_bot              = db.Column(db.Integer, nullable=False)
+	pref_maintainer       = db.Column(db.Integer, nullable=False)
+	pref_editor_alert     = db.Column(db.Integer, nullable=False)
+	pref_editor_misc      = db.Column(db.Integer, nullable=False)
+	pref_other            = db.Column(db.Integer, nullable=False)
+
+	def __init__(self, user):
+		self.user = user
+		self.pref_package_edit = 1
+		self.pref_package_approval = 1
+		self.pref_new_thread = 1
+		self.pref_new_review = 1
+		self.pref_thread_reply = 2
+		self.pref_bot = 1
+		self.pref_maintainer = 1
+		self.pref_editor_alert = 1
+		self.pref_editor_misc = 0
+		self.pref_other = 0
+
+	def get_can_email(self, notification_type):
+		return getattr(self, "pref_" + notification_type.toName()) == 2
+
+	def set_can_email(self, notification_type, value):
+		value = 2 if value else 0
+		setattr(self, "pref_" + notification_type.toName(), value)
+
+	def get_can_digest(self, notification_type):
+		return getattr(self, "pref_" + notification_type.toName()) >= 1
+
+	def set_can_digest(self, notification_type, value):
+		if self.get_can_email(notification_type):
+			return
+
+		value = 1 if value else 0
+		setattr(self, "pref_" + notification_type.toName(), value)

app/public/static/notification.svg

@@ -1,93 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<!-- Created with Inkscape (http://www.inkscape.org/) -->
-
-<svg
-   xmlns:dc="http://purl.org/dc/elements/1.1/"
-   xmlns:cc="http://creativecommons.org/ns#"
-   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
-   xmlns:svg="http://www.w3.org/2000/svg"
-   xmlns="http://www.w3.org/2000/svg"
-   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
-   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
-   width="31.412476mm"
-   height="28.42704mm"
-   viewBox="0 0 31.412476 28.42704"
-   version="1.1"
-   id="svg8"
-   inkscape:version="0.92.2 2405546, 2018-03-11"
-   sodipodi:docname="notification.svg">
-  <defs
-     id="defs2" />
-  <sodipodi:namedview
-     id="base"
-     pagecolor="#ffffff"
-     bordercolor="#666666"
-     borderopacity="1.0"
-     inkscape:pageopacity="0.0"
-     inkscape:pageshadow="2"
-     inkscape:zoom="3.959798"
-     inkscape:cx="47.783055"
-     inkscape:cy="53.095002"
-     inkscape:document-units="mm"
-     inkscape:current-layer="g821"
-     showgrid="false"
-     showguides="true"
-     inkscape:guide-bbox="true"
-     inkscape:window-width="1877"
-     inkscape:window-height="1080"
-     inkscape:window-x="43"
-     inkscape:window-y="0"
-     inkscape:window-maximized="1"
-     inkscape:snap-nodes="false"
-     inkscape:snap-global="false">
-    <sodipodi:guide
-       position="15.708731,30.13464"
-       orientation="1,0"
-       id="guide823"
-       inkscape:locked="false" />
-    <sodipodi:guide
-       position="13.971479,23.252449"
-       orientation="0,1"
-       id="guide825"
-       inkscape:locked="false" />
-  </sodipodi:namedview>
-  <metadata
-     id="metadata5">
-    <rdf:RDF>
-      <cc:Work
-         rdf:about="">
-        <dc:format>image/svg+xml</dc:format>
-        <dc:type
-           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
-        <dc:title></dc:title>
-      </cc:Work>
-    </rdf:RDF>
-  </metadata>
-  <g
-     inkscape:label="Layer 1"
-     inkscape:groupmode="layer"
-     id="layer1"
-     transform="translate(-10.149596,-260.42124)">
-    <g
-       id="g821"
-       transform="translate(4.6772167)">
-      <path
-         style="fill:#cccccc;stroke:none;stroke-width:0.22791502px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
-         d="m 34.706713,281.95262 c 0,0 0.05088,-0.47786 -0.454138,-0.79248 -0.598885,-0.37309 -1.792456,-0.65695 -2.135937,-1.62493 -0.633129,-1.78428 -1.208702,-12.36663 -3.568549,-14.5538 -2.359846,-2.18717 -7.194655,-4.17785 -7.194655,-4.17785 h -0.345341 c 0,0 -4.834807,1.99068 -7.194655,4.17785 -2.359846,2.18717 -2.935418,12.76952 -3.568547,14.5538 -0.3187212,0.89821 -1.5484903,1.15935 -2.1196334,1.54022 -0.5634184,0.37571 -0.4704434,0.87719 -0.4704434,0.87719 z"
-         id="path817"
-         inkscape:connector-curvature="0"
-         sodipodi:nodetypes="csssccssscc" />
-      <path
-         style="fill:#cccccc;fill-rule:evenodd;stroke:none;stroke-width:0.32969889"
-         id="path841"
-         sodipodi:type="arc"
-         sodipodi:cx="21.295879"
-         sodipodi:cy="283.65527"
-         sodipodi:rx="4.158649"
-         sodipodi:ry="5.2277269"
-         sodipodi:start="0"
-         sodipodi:end="3.1415927"
-         d="m 25.454528,283.65527 a 4.158649,5.2277269 0 0 1 -2.079324,4.52735 4.158649,5.2277269 0 0 1 -4.158649,0 4.158649,5.2277269 0 0 1 -2.079325,-4.52735 l 4.158649,0 z" />
-    </g>
-  </g>
-</svg>

app/public/static/notification_alert.svg

@@ -1,93 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<!-- Created with Inkscape (http://www.inkscape.org/) -->
-
-<svg
-   xmlns:dc="http://purl.org/dc/elements/1.1/"
-   xmlns:cc="http://creativecommons.org/ns#"
-   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
-   xmlns:svg="http://www.w3.org/2000/svg"
-   xmlns="http://www.w3.org/2000/svg"
-   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
-   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
-   width="31.412476mm"
-   height="28.42704mm"
-   viewBox="0 0 31.412476 28.42704"
-   version="1.1"
-   id="svg8"
-   inkscape:version="0.92.2 2405546, 2018-03-11"
-   sodipodi:docname="notification_alert.svg">
-  <defs
-     id="defs2" />
-  <sodipodi:namedview
-     id="base"
-     pagecolor="#ffffff"
-     bordercolor="#666666"
-     borderopacity="1.0"
-     inkscape:pageopacity="0.0"
-     inkscape:pageshadow="2"
-     inkscape:zoom="3.959798"
-     inkscape:cx="47.783055"
-     inkscape:cy="63.196527"
-     inkscape:document-units="mm"
-     inkscape:current-layer="g821"
-     showgrid="false"
-     showguides="true"
-     inkscape:guide-bbox="true"
-     inkscape:window-width="1877"
-     inkscape:window-height="1080"
-     inkscape:window-x="43"
-     inkscape:window-y="0"
-     inkscape:window-maximized="1"
-     inkscape:snap-nodes="false"
-     inkscape:snap-global="false">
-    <sodipodi:guide
-       position="15.708731,30.13464"
-       orientation="1,0"
-       id="guide823"
-       inkscape:locked="false" />
-    <sodipodi:guide
-       position="13.971479,23.252449"
-       orientation="0,1"
-       id="guide825"
-       inkscape:locked="false" />
-  </sodipodi:namedview>
-  <metadata
-     id="metadata5">
-    <rdf:RDF>
-      <cc:Work
-         rdf:about="">
-        <dc:format>image/svg+xml</dc:format>
-        <dc:type
-           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
-        <dc:title></dc:title>
-      </cc:Work>
-    </rdf:RDF>
-  </metadata>
-  <g
-     inkscape:label="Layer 1"
-     inkscape:groupmode="layer"
-     id="layer1"
-     transform="translate(-10.149596,-260.42124)">
-    <g
-       id="g821"
-       transform="translate(4.6772167)">
-      <path
-         style="fill:#ffff00;stroke:none;stroke-width:0.22791502px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
-         d="m 34.706713,281.95262 c 0,0 0.05088,-0.47786 -0.454138,-0.79248 -0.598885,-0.37309 -1.792456,-0.65695 -2.135937,-1.62493 -0.633129,-1.78428 -1.208702,-12.36663 -3.568549,-14.5538 -2.359846,-2.18717 -7.194655,-4.17785 -7.194655,-4.17785 h -0.345341 c 0,0 -4.834807,1.99068 -7.194655,4.17785 -2.359846,2.18717 -2.935418,12.76952 -3.568547,14.5538 -0.3187212,0.89821 -1.5484903,1.15935 -2.1196334,1.54022 -0.5634184,0.37571 -0.4704434,0.87719 -0.4704434,0.87719 z"
-         id="path817"
-         inkscape:connector-curvature="0"
-         sodipodi:nodetypes="csssccssscc" />
-      <path
-         style="fill:#ffff00;fill-rule:evenodd;stroke:none;stroke-width:0.32969889"
-         id="path841"
-         sodipodi:type="arc"
-         sodipodi:cx="21.295879"
-         sodipodi:cy="283.65527"
-         sodipodi:rx="4.158649"
-         sodipodi:ry="5.2277269"
-         sodipodi:start="0"
-         sodipodi:end="3.1415927"
-         d="m 25.454528,283.65527 a 4.158649,5.2277269 0 0 1 -2.079324,4.52735 4.158649,5.2277269 0 0 1 -4.158649,0 4.158649,5.2277269 0 0 1 -2.079325,-4.52735 l 4.158649,0 z" />
-    </g>
-  </g>
-</svg>

app/public/static/package_create.js

@@ -22,7 +22,7 @@ $(function() {
 
 			function setField(id, value) {
 				if (value && value != "") {
-					var ele = $(id);
+					const ele = $(id);
 					ele.val(value);
 					ele.trigger("change");
 				}
@@ -30,14 +30,13 @@ $(function() {
 
 			performTask("/tasks/getmeta/new/?url=" + encodeURI(repoURL)).then(function(result) {
 				setField("#name", result.name);
-				setField("#provides_str", result.provides);
 				setField("#title", result.title);
 				setField("#repo", result.repo || repoURL);
 				setField("#issueTracker", result.issueTracker);
 				setField("#desc", result.description);
 				setField("#short_desc", result.short_description);
-				setField("#harddep_str", result.depends);
-				setField("#softdep_str", result.optional_depends);
+				// setField("#harddep_str", result.depends);
+				// setField("#softdep_str", result.optional_depends);
 				setField("#short_desc", result.short_description);
 				setField("#forums", result.forumId);
 				if (result.type && result.type.length > 2) {

app/public/static/package_edit.js

@@ -11,8 +11,8 @@ $(function() {
 
 	$("#forums").on('paste', function(e) {
 		try {
-			var pasteData = e.originalEvent.clipboardData.getData('text')
-			var url = new URL(pasteData);
+			const pasteData = e.originalEvent.clipboardData.getData('text');
+			const url = new URL(pasteData);
 			if (url.hostname == "forum.minetest.net") {
 				$(this).val(url.searchParams.get("t"));
 				e.preventDefault();
@@ -22,6 +22,10 @@ $(function() {
 		}
 	});
 
+	$("#forums-button").click(function(e) {
+		window.open("https://forum.minetest.net/viewtopic.php?t=" + $("#forums").val(), "_blank");
+	});
+
 	let hint = null;
 	function showHint(ele, text) {
 		if (hint) {
@@ -42,7 +46,7 @@ $(function() {
 		there's no need to use phrases such as \"adds X to the game\".`
 
 	$("#short_desc").on("change paste keyup", function() {
-		var val = $(this).val().toLowerCase();
+		const val = $(this).val().toLowerCase();
 		if (val.indexOf("minetest") >= 0 || val.indexOf("mod") >= 0 ||
 				val.indexOf("modpack") >= 0 || val.indexOf("mod pack") >= 0) {
 			showHint($(this), hint_mtmods);
@@ -54,9 +58,9 @@ $(function() {
 		}
 	})
 
-	var btn = $("#forums").parent().find("label").append("<a class='ml-3 btn btn-sm btn-primary'>Open</a>");
+	const btn = $("#forums").parent().find("label").append("<a class='ml-3 btn btn-sm btn-primary'>Open</a>");
 	btn.click(function() {
-		var id = $("#forums").val();
+		const id = $("#forums").val();
 		if (/^\d+$/.test(id)) {
 			window.open("https://forum.minetest.net/viewtopic.php?t=" + id, "_blank");
 		}

app/public/static/polltask.js

@@ -19,7 +19,8 @@ function getJSON(url, method) {
 
 function pollTask(poll_url, disableTimeout) {
 	return new Promise(function(resolve, reject) {
-		var tries = 0;
+		let tries = 0;
+
 		function retry() {
 			tries++;
 			if (!disableTimeout && tries > 30) {

app/public/static/release_minmax.js

@@ -1,11 +1,11 @@
-var min = $("#min_rel");
-var max = $("#max_rel");
-var none = $("#min_rel option:first-child").attr("value");
-var warning = $("#minmax_warning");
+const min = $("#min_rel");
+const max = $("#max_rel");
+const none = $("#min_rel option:first-child").attr("value");
+const warning = $("#minmax_warning");
 
 function ver_check() {
-	var minv = min.val();
-	var maxv = max.val();
+	const minv = min.val();
+	const maxv = max.val();
 
 	if (minv != none && maxv != none && minv > maxv) {
 		warning.show();

app/public/static/tagselector.js

@@ -6,115 +6,115 @@
  */
 (function($) {
 	function hide_error(input) {
-		var err = input.parent().parent().find(".invalid-remaining");
+		const err = input.parent().parent().find(".invalid-remaining");
 		err.hide();
 	}
 
 	function show_error(input, msg) {
-		var err = input.parent().parent().find(".invalid-remaining");
+		const err = input.parent().parent().find(".invalid-remaining");
 		console.log(err.length);
 		err.text(msg);
 		err.show();
 	}
 
-	$.fn.selectSelector = function(source, name, select) {
+	$.fn.selectSelector = function(source, select) {
 		return this.each(function() {
-				var selector = $(this),
-					input = $('input[type=text]', this);
+			const selector = $(this),
+				input = $('input[type=text]', this);
 
-				selector.click(function() { input.focus(); })
-					.delegate('.badge a', 'click', function() {
-						var id = $(this).parent().data("id");
-						for (var i = 0; i < source.length; i++) {
-							if (source[i].id == id) {
-								source[i].selected = null;
-							}
-						}
-						select.find("option[value=" + id + "]").attr("selected", null)
-						recreate();
-					});
+			selector.click(function() { input.focus(); })
+				.delegate('.badge a', 'click', function() {
+					const id = $(this).parent().data("id");
+					select.find("option[value=" + id + "]").attr("selected", false)
+					recreate();
+				});
 
-				function addTag(item) {
-					var tag = $('<span class="badge badge-pill badge-primary"/>')
-						.text(item.toString() + ' ')
-						.data("id", item.id)
-						.append('<a>x</a>')
-						.insertBefore(input);
-					input.attr("placeholder", null);
-					select.find("option[value=" + item.id + "]").attr("selected", "selected")
-					hide_error(input);
+			function addTag(id, text) {
+				const idx = text.indexOf(':');
+				if (idx > 0) {
+					text = text.substr(0, idx);
 				}
 
-				function recreate() {
-					selector.find("span").remove();
-					for (var i = 0; i < source.length; i++) {
-						if (source[i].selected) {
-							addTag(source[i]);
-						}
-					}
-				}
-				recreate();
+				$('<span class="badge badge-pill badge-primary"/>')
+					.text(text + ' ')
+					.data("id", id)
+					.append('<a>x</a>')
+					.insertBefore(input);
+				input.attr("placeholder", null);
+				select.find("option[value='" + id + "']").attr("selected", "selected")
+				hide_error(input);
+			}
 
-				input.focusout(function(e) {
-					var value = input.val().trim()
-					if (value != "") {
-						show_error(input, "Please select an existing tag, it;s not possible to add custom ones.");
+			function recreate() {
+				selector.find("span").remove();
+				select.find("option").each(function() {
+					if (this.hasAttribute("selected")) {
+						addTag(this.getAttribute("value"), this.innerText);
 					}
+				});
+			}
+			recreate();
+
+			input.focusout(function() {
+				const value = input.val().trim();
+				if (value != "") {
+					show_error(input, "Please select an existing tag, it;s not possible to add custom ones.");
+				}
+			})
+
+			input.keydown(function(e) {
+					if (e.keyCode === $.ui.keyCode.TAB && $(this).data('ui-autocomplete').menu.active)
+						e.preventDefault();
 				})
+				.autocomplete({
+					minLength: 0,
+					source: source,
+					select: function(event, ui) {
+						addTag(ui.item.id, ui.item.toString());
+						input.val("");
+						return false;
+					}
+				}).focus(function() {
+					// The following works only once.
+					// $(this).trigger('keydown.autocomplete');
+					// As suggested by digitalPBK, works multiple times
+					// $(this).data("autocomplete").search($(this).val());
+					// As noted by Jonny in his answer, with newer versions use uiAutocomplete
+					$(this).data("ui-autocomplete").search($(this).val());
+				});
 
-				input.keydown(function(e) {
-						if (e.keyCode === $.ui.keyCode.TAB && $(this).data('ui-autocomplete').menu.active)
-							e.preventDefault();
-					})
-					.autocomplete({
-						minLength: 0,
-						source: source,
-						select: function(event, ui) {
-							addTag(ui.item);
-							input.val("");
-							return false;
-						}
-					}).focus(function() {
-						// The following works only once.
-						// $(this).trigger('keydown.autocomplete');
-						// As suggested by digitalPBK, works multiple times
-						// $(this).data("autocomplete").search($(this).val());
-						// As noted by Jonny in his answer, with newer versions use uiAutocomplete
-						$(this).data("ui-autocomplete").search($(this).val());
-					});
+			input.data('ui-autocomplete')._renderItem = function(ul, item) {
+					return $('<li/>')
+						.data('item.autocomplete', item)
+						.append($('<a/>').text(item.toString()))
+						.appendTo(ul);
+				};
 
-				input.data('ui-autocomplete')._renderItem = function(ul, item) {
-						return $('<li/>')
-							.data('item.autocomplete', item)
-							.append($('<a/>').text(item.toString()))
-							.appendTo(ul);
-					};
-
-				input.data('ui-autocomplete')._resizeMenu = function(ul, item) {
-						var ul = this.menu.element;
-						ul.outerWidth(Math.max(
-							ul.width('').outerWidth(),
-							selector.outerWidth()
-						));
-					};
-			});
+			input.data('ui-autocomplete')._resizeMenu = function(ul, item) {
+					var ul = this.menu.element;
+					ul.outerWidth(Math.max(
+						ul.width('').outerWidth(),
+						selector.outerWidth()
+					));
+				};
+		});
 	}
 
 	$.fn.csvSelector = function(source, name, result, allowSlash) {
 		return this.each(function() {
-				var selector = $(this),
-					input = $('input[type=text]', this);
+			const selector = $(this),
+				input = $('input[type=text]', this);
 
-				var selected = [];
-				var lookup = {};
-				for (var i = 0; i < source.length; i++) {
+			let selected = [];
+			const lookup = {};
+			for (var i = 0; i < source.length; i++) {
 					lookup[source[i].id] = source[i];
 				}
 
 				selector.click(function() { input.focus(); })
 					.delegate('.badge a', 'click', function() {
-						var id = $(this).parent().data("id");
-						for (var i = 0; i < selected.length; i++) {
+						const id = $(this).parent().data("id");
+						for (let i = 0; i < selected.length; i++) {
 							if (selected[i] == id) {
 								selected.splice(i, 1);
 							}
@@ -123,7 +123,7 @@
 					});
 
 				function selectItem(id) {
-					for (var i = 0; i < selected.length; i++) {
+					for (let i = 0; i < selected.length; i++) {
 						if (selected[i] == id) {
 							return false;
 						}
@@ -133,7 +133,7 @@
 				}
 
 				function addTag(id, value) {
-					var tag = $('<span class="badge badge-pill badge-primary"/>')
+					const tag = $('<span class="badge badge-pill badge-primary"/>')
 						.text(value)
 						.data("id", id)
 						.append(' <a>x</a>')
@@ -145,8 +145,8 @@
 
 				function recreate() {
 					selector.find("span").remove();
-					for (var i = 0; i < selected.length; i++) {
-						var value = lookup[selected[i]] || { value: selected[i] };
+					for (let i = 0; i < selected.length; i++) {
+						const value = lookup[selected[i]] || {value: selected[i]};
 						addTag(selected[i], value.value);
 					}
 					result.val(selected.join(","))
@@ -154,9 +154,9 @@
 
 				function readFromResult() {
 					selected = [];
-					var selected_raw = result.val().split(",");
-					for (var i = 0; i < selected_raw.length; i++) {
-						var raw = selected_raw[i].trim();
+					const selected_raw = result.val().split(",");
+					for (let i = 0; i < selected_raw.length; i++) {
+						const raw = selected_raw[i].trim();
 						if (lookup[raw] || raw.match(/^([a-z0-9_]+)$/)) {
 							selected.push(raw);
 						}
@@ -169,7 +169,7 @@
 				result.change(readFromResult);
 
 				input.focusout(function() {
-					var item = input.val();
+					const item = input.val();
 					if (item.length == 0) {
 						input.data("ui-autocomplete").search("");
 					} else if (item.match(/^([a-z0-9_]+)$/)) {
@@ -238,33 +238,32 @@
 
 	$(function() {
 		$(".multichoice_selector").each(function() {
-			var ele = $(this);
-			var sel = ele.parent().find("select");
+			const ele = $(this);
+			const sel = ele.parent().find("select");
 			sel.hide();
 
-			var options = [];
+			const options = [];
 			sel.find("option").each(function() {
-				var text = $(this).text();
+				const text = $(this).text();
 				options.push({
 					id: $(this).attr("value"),
 					value: text,
-					selected: $(this).attr("selected") ? true : false,
+					selected: !!$(this).attr("selected"),
 					toString: function() { return text; },
 				});
 			});
 
-			console.log(options);
-			ele.selectSelector(options, sel.attr("name"), sel);
+			ele.selectSelector(options, sel);
 		});
 
 		$(".metapackage_selector").each(function() {
-			var input = $(this).parent().children("input[type='text']");
+			const input = $(this).parent().children("input[type='text']");
 			input.hide();
 			$(this).csvSelector(meta_packages, input.attr("name"), input);
 		});
 
 		$(".deps_selector").each(function() {
-			var input = $(this).parent().children("input[type='text']");
+			const input = $(this).parent().children("input[type='text']");
 			input.hide();
 			$(this).csvSelector(all_packages, input.attr("name"), input);
 		});

app/public/static/topic_discard.js

@@ -1,7 +1,7 @@
 $(".topic-discard").click(function() {
-	var ele = $(this);
-	var tid = ele.attr("data-tid");
-	var discard = !ele.parent().parent().hasClass("discardtopic");
+	const ele = $(this);
+	const tid = ele.attr("data-tid");
+	const discard = !ele.parent().parent().hasClass("discardtopic");
 	fetch(new Request("/api/topic_discard/?tid=" + tid +
 			"&discard=" + (discard ? "true" : "false"), {
 		method: "post",

app/querybuilder.py

@@ -1,8 +1,11 @@
-from .models import db, PackageType, Package, ForumTopic, License, MinetestRelease, PackageRelease
-from .utils import isNo, isYes
-from sqlalchemy.sql.expression import func
-from flask import abort
+from flask import abort, current_app
 from sqlalchemy import or_
+from sqlalchemy.orm import subqueryload
+from sqlalchemy.sql.expression import func
+
+from .models import db, PackageType, Package, ForumTopic, License, MinetestRelease, PackageRelease, User, Tag, ContentWarning, PackageState
+from .utils import isYes, get_int_or_abort
+
 
 class QueryBuilder:
 	title  = None
@@ -19,18 +22,39 @@ class QueryBuilder:
 		if len(types) > 0:
 			title = ", ".join([type.value + "s" for type in types])
 
+		# Get tags types
+		tags = args.getlist("tag")
+		tags = [Tag.query.filter_by(name=tname).first() for tname in tags]
+		tags = [tag for tag in tags if tag is not None]
+
+		# Hide
 		hide_flags = args.getlist("hide")
 
+
 		self.title  = title
 		self.types  = types
-		self.search = args.get("q")
+		self.tags   = tags
+
 		self.random = "random" in args
 		self.lucky  = "lucky" in args
-		self.hide_nonfree = "nonfree" in hide_flags
 		self.limit  = 1 if self.lucky else None
 		self.order_by  = args.get("sort")
 		self.order_dir = args.get("order") or "desc"
-		self.protocol_version = args.get("protocol_version")
+
+		self.hide_nonfree = "nonfree" in hide_flags
+		self.hide_flags = set(hide_flags)
+		self.hide_flags.discard("nonfree")
+
+		# Filters
+		self.search = args.get("q")
+		self.author = args.get("author")
+
+		protocol_version = get_int_or_abort(args.get("protocol_version"))
+		minetest_version = args.get("engine_version")
+		if protocol_version or minetest_version:
+			self.version = MinetestRelease.get(minetest_version, protocol_version)
+		else:
+			self.version = None
 
 		self.show_discarded = isYes(args.get("show_discarded"))
 		self.show_added = args.get("show_added")
@@ -40,41 +64,120 @@ class QueryBuilder:
 		if self.search is not None and self.search.strip() == "":
 			self.search = None
 
-	def setSortIfNone(self, name):
+	def setSortIfNone(self, name, dir="desc"):
 		if self.order_by is None:
 			self.order_by = name
+			self.order_dir = dir
 
-	def getMinetestVersion(self):
-		if not self.protocol_version:
-			return None
+	def getReleases(self):
+		releases_query = db.session.query(PackageRelease.package_id, func.max(PackageRelease.id)) \
+			.select_from(PackageRelease).filter(PackageRelease.approved) \
+			.group_by(PackageRelease.package_id)
 
-		self.protocol_version = int(self.protocol_version)
-		version = MinetestRelease.query.filter(MinetestRelease.protocol>=self.protocol_version).first()
-		if version is not None:
-			return version.id
-		else:
-			return 10000000
+		if self.version:
+			releases_query = releases_query \
+				.filter(or_(PackageRelease.min_rel_id == None,
+					PackageRelease.min_rel_id <= self.version.id)) \
+				.filter(or_(PackageRelease.max_rel_id == None,
+					PackageRelease.max_rel_id >= self.version.id))
+
+		return releases_query.all()
+
+	def convertToDictionary(self, packages):
+		releases = {}
+		for [package_id, release_id] in self.getReleases():
+			releases[package_id] = release_id
+
+		def toJson(package: Package):
+			release_id = releases.get(package.id)
+			return package.getAsDictionaryShort(current_app.config["BASE_URL"], release_id=release_id, no_load=True)
+
+		return [toJson(pkg) for pkg in packages]
 
 	def buildPackageQuery(self):
-		query = Package.query.filter_by(soft_deleted=False, approved=True)
+		if self.order_by == "last_release":
+			query = db.session.query(Package).select_from(PackageRelease).join(Package) \
+				.filter_by(state=PackageState.APPROVED)
+		else:
+			query = Package.query.filter_by(state=PackageState.APPROVED)
 
+		query = query.options(subqueryload(Package.main_screenshot))
+
+		query = self.orderPackageQuery(self.filterPackageQuery(query))
+
+		if self.limit:
+			query = query.limit(self.limit)
+
+		return query
+
+	def filterPackageQuery(self, query):
 		if len(self.types) > 0:
 			query = query.filter(Package.type.in_(self.types))
 
+		if self.author:
+			author = User.query.filter_by(username=self.author).first()
+			if not author:
+				abort(404)
+
+			query = query.filter_by(author=author)
+
+		for tag in self.tags:
+			query = query.filter(Package.tags.any(Tag.id == tag.id))
+
+		if "android_default" in self.hide_flags:
+			query = query.filter(~ Package.content_warnings.any())
+		else:
+			for flag in self.hide_flags:
+				warning = ContentWarning.query.filter_by(name=flag).first()
+				if warning:
+					query = query.filter(~ Package.content_warnings.any(ContentWarning.id == warning.id))
+
+		if self.hide_nonfree:
+			query = query.filter(Package.license.has(License.is_foss == True))
+			query = query.filter(Package.media_license.has(License.is_foss == True))
+
+		if self.version:
+			query = query.join(Package.releases) \
+				.filter(PackageRelease.approved == True) \
+				.filter(or_(PackageRelease.min_rel_id == None,
+					PackageRelease.min_rel_id <= self.version.id)) \
+				.filter(or_(PackageRelease.max_rel_id == None,
+					PackageRelease.max_rel_id >= self.version.id))
+
+		return query
+
+	def orderPackageQuery(self, query):
 		if self.search:
-			query = query.search(self.search, sort=True)
+			query = query.search(self.search, sort=self.order_by is None)
 
 		if self.random:
 			query = query.order_by(func.random())
-		else:
-			to_order = None
-			if self.order_by is None or self.order_by == "score":
-				to_order = Package.score
-			elif self.order_by == "created_at":
-				to_order = Package.created_at
-			else:
-				abort(400)
+			return query
 
+		to_order = None
+		if self.order_by is None and self.search:
+			pass
+		elif self.order_by is None or self.order_by == "score":
+			to_order = Package.score
+		elif self.order_by == "reviews":
+			query = query.filter(Package.reviews.any())
+			to_order = (Package.score - Package.score_downloads)
+		elif self.order_by == "name":
+			to_order = Package.name
+		elif self.order_by == "title":
+			to_order = Package.title
+		elif self.order_by == "downloads":
+			to_order = Package.downloads
+		elif self.order_by == "created_at" or self.order_by == "date":
+			to_order = Package.created_at
+		elif self.order_by == "approved_at" or self.order_by == "date":
+			to_order = Package.approved_at
+		elif self.order_by == "last_release":
+			to_order = PackageRelease.releaseDate
+		else:
+			abort(400)
+
+		if to_order is not None:
 			if self.order_dir == "asc":
 				to_order = db.asc(to_order)
 			elif self.order_dir == "desc":
@@ -84,20 +187,6 @@ class QueryBuilder:
 
 			query = query.order_by(to_order)
 
-		if self.hide_nonfree:
-			query = query.filter(Package.license.has(License.is_foss == True))
-			query = query.filter(Package.media_license.has(License.is_foss == True))
-
-		if self.protocol_version:
-			version = self.getMinetestVersion()
-			query = query.join(Package.releases) \
-				.filter(PackageRelease.approved==True) \
-				.filter(or_(PackageRelease.min_rel_id==None, PackageRelease.min_rel_id<=version)) \
-				.filter(or_(PackageRelease.max_rel_id==None, PackageRelease.max_rel_id>=version))
-
-		if self.limit:
-			query = query.limit(self.limit)
-
 		return query
 
 	def buildTopicQuery(self, show_added=False):
@@ -114,9 +203,8 @@ class QueryBuilder:
 			query = query.order_by(db.asc(ForumTopic.wip), db.asc(ForumTopic.name), db.asc(ForumTopic.title))
 		elif self.order_by == "views":
 			query = query.order_by(db.desc(ForumTopic.views))
-		elif self.order_by == "date":
+		elif self.order_by == "created_at" or self.order_by == "date":
 			query = query.order_by(db.asc(ForumTopic.created_at))
-			sort_by = "date"
 
 		if self.search:
 			query = query.filter(ForumTopic.title.ilike('%' + self.search + '%'))

app/rediscache.py

@@ -4,7 +4,7 @@ from . import r
 # and also means that the releases code avoids knowing about `app`
 
 def make_download_key(ip, package):
-	return ("{}/{}/{}").format(ip, package.author.username, package.name)
+	return "{}/{}/{}".format(ip, package.author.username, package.name)
 
 def set_key(key, v):
 	r.set(key, v)

app/sass.py

@@ -53,11 +53,11 @@ def sass(app, inputDir='scss', outputPath='static', force=False, cacheDir="publi
 		cacheFile = "%s/%s.css" % (cacheDir, filepath)
 
 		# Source file exists, and needs regenerating
-		if os.path.isfile(sassfile) and (force or not os.path.isfile(cacheFile) or \
-				os.path.getmtime(sassfile) > os.path.getmtime(cacheFile)):
+		if os.path.isfile(sassfile) and (force or not os.path.isfile(cacheFile) or
+										 os.path.getmtime(sassfile) > os.path.getmtime(cacheFile)):
 			_convert(inputDir, sassfile, cacheFile)
 			app.logger.debug('Compiled %s into %s' % (sassfile, cacheFile))
 
 		return send_from_directory(cacheDir, filepath + ".css")
 
-	app.add_url_rule("/%s/<path:filepath>.css" % (outputPath), 'sass', _sass)
+	app.add_url_rule("/%s/<path:filepath>.css" % outputPath, 'sass', _sass)

app/scss/comments.scss

@@ -26,4 +26,10 @@
 			border-width: 14px;
 		}
 	}
+
+	.user-photo {
+		width: 60px;
+		height: 60px;
+		object-fit: cover;
+	}
 }

app/scss/components.scss

@@ -1,7 +1,7 @@
 .ui-autocomplete, ui-front {
 	position:absolute;
 	cursor:default;
-	z-index:1001 !important
+	z-index:1100 !important
 }
 
 .ui-autocomplete {
@@ -32,14 +32,14 @@
 .bulletselector input {
 	border: none;
 	border-radius: 0;
-    -moz-border-radius: 0;
+	-moz-border-radius: 0;
 	box-shadow: none;
 	-moz-box-shadow: none;
 	-webkit-box-shadow: none;
 	width: auto;
 	min-width: 50px;
 	float: left;
-    padding: 4px 0;
+	padding: 4px 0;
 	white-space: nowrap;
 	background: transparent;
 }
@@ -81,6 +81,10 @@
 	color: #2c2 !important;
 }
 
+.BOT a, .BOT {
+	color: #FFDF00 !important;
+}
+
 .wiptopic a:not(.btn) {
 	color: #7ac;
 }
@@ -122,3 +126,86 @@
 	background-color: #222 !important;
 	color: white !important;
 }
+
+.sortable {
+	user-select: none;
+}
+
+blockquote {
+	padding: 0.5em 1em;
+	border-left: 3px solid rgba(255, 255, 255, 0.25);
+
+	& > *:last-child {
+		margin-bottom: 0;
+	}
+}
+
+.tabs-container {
+	background: #1c1c1c;
+	border-bottom: 1px solid #444;
+
+	.nav {
+		border-bottom: none;
+	}
+}
+
+.toc {
+	.nav-link {
+		color: #ADADAD;
+		padding: 0.25rem 0.5rem;
+
+		&:hover, &.active {
+			color: #DDD;
+		}
+	}
+
+	.nav .nav {
+		margin: 0.1em 0 0.1em 0.7rem;
+		padding-left: 0.25em;
+		border-left: 3px solid rgba(173, 173, 173, 0.25);
+	}
+
+	& > .nav > * > .nav {
+		margin-bottom: 0.5em;
+	}
+}
+
+
+.signin {
+	width: 100%;
+	max-width: 330px;
+	padding: 15px;
+	margin: 0 auto;
+
+  	.form-control {
+		position: relative;
+		box-sizing: border-box;
+		height: auto;
+		padding: 10px;
+		font-size: 16px;
+	}
+
+	.form-group {
+		margin-bottom: 0;
+	}
+
+	.form-control:focus {
+		z-index: 2;
+	}
+
+	.checkbox {
+		font-weight: 400;
+	}
+
+	input[type="text"] {
+		margin-bottom: -1px;
+		border-bottom-right-radius: 0;
+		border-bottom-left-radius: 0;
+	}
+
+	input[type="password"] {
+		margin-bottom: 10px;
+		border-top-left-radius: 0;
+		border-top-right-radius: 0;
+	}
+}

app/scss/custom.scss

@@ -3,12 +3,81 @@
 @import "packagegrid.scss";
 @import "comments.scss";
 
+footer {
+	text-align: center;
+	color: #999;
+
+	a {
+		color: #2e997e;
+	}
+
+	a:hover {
+		color: #00bc8c;
+	}
+}
+
+h1 {
+	font-size: 2em;
+	font-weight: bold;
+	margin: 0 0 0.5em;
+	letter-spacing: .05em
+}
+
+h2 {
+	font-size: 1.8em;
+	font-weight: bold;
+	color: white;
+	margin: 1.5em 0 1em;
+	letter-spacing: .05em;
+	padding: 0 0 0.5em 0;
+	border-bottom: 1px solid #444;
+}
+
+h3 {
+	font-size: 1.3em;
+	font-weight: bold;
+	color: white;
+	margin: 1.5em 0 1em;
+	letter-spacing: .05em
+}
+
+.badge-notify {
+	background:yellow; /* #00bc8c;*/
+	color: black;
+	position:relative;
+	top: -12px;
+	left: -10px;
+	margin-right: -10px;
+	font-size:10px;
+}
+
+a:hover .badge-notify {
+	color: black;
+}
+
+p, .content li {
+	-moz-osx-font-smoothing: grayscale;
+	-webkit-font-smoothing: antialiased !important;
+	-moz-font-smoothing: antialiased !important;
+	text-rendering: optimizelegibility !important;
+	letter-spacing: .03em;
+	line-height: 1.6em;
+}
+
+pre code {
+	display: block;
+	border: 1px solid rgba(255, 255, 255, 0.1);
+	background: rgba(51, 51, 51, 0.25);
+	padding: 0.75rem 1.25rem;
+	border-radius: 0.25rem;
+}
+
 .dropdown-menu {
 	margin-top: 0;
 }
 
 .dropdown:hover .dropdown-menu {
-   display: block;
+	display: block;
 }
 
 .nav-link > img {
@@ -57,8 +126,15 @@
 	text-decoration: none;
 }
 
-.card .table {
-	margin-bottom: 0;
+.card {
+	.card-header {
+		margin: 0;
+		font-size: 100%;
+		font-weight: normal;
+	}
+	.table {
+		margin-bottom: 0;
+	}
 }
 
 .btn-download {
@@ -71,3 +147,15 @@
 	-webkit-box-shadow: 0 0 0 0.2rem rgba(231, 76, 60, 0.5);
 	box-shadow: 0 0 0 0.2rem rgba(231, 76, 60, 0.5);
 }
+
+.ranks-table tr {
+	th, td {
+		text-align: center;
+	}
+
+	td:first-child, th:first-child {
+		text-align: left;
+	}
+}
+
+@import "dracula.scss";

app/scss/dracula.scss

@@ -0,0 +1,94 @@
+/* Dracula Theme v1.2.5
+ *
+ * https://github.com/zenorocha/dracula-theme
+ *
+ * Copyright 2016, All rights reserved
+ *
+ * Code licensed under the MIT license
+ * http://zenorocha.mit-license.org
+ *
+ * @author Rob G <wowmotty@gmail.com>
+ * @author Chris Bracco <chris@cbracco.me>
+ * @author Zeno Rocha <hi@zenorocha.com>
+ */
+
+.highlight, .codehilite {
+	color: #f8f8f2;
+
+	.hll { background-color: #f1fa8c }
+	.c { color: #6272a4 } /* Comment */
+	.err { color: #f8f8f2 } /* Error */
+	.g { color: #f8f8f2 } /* Generic */
+	.k { color: #ff79c6 } /* Keyword */
+	.l { color: #f8f8f2 } /* Literal */
+	.n { color: #f8f8f2 } /* Name */
+	.o { color: #ff79c6 } /* Operator */
+	.x { color: #f8f8f2 } /* Other */
+	.p { color: #f8f8f2 } /* Punctuation */
+	.ch { color: #6272a4 } /* Comment.Hashbang */
+	.cm { color: #6272a4 } /* Comment.Multiline */
+	.cp { color: #ff79c6 } /* Comment.Preproc */
+	.cpf { color: #6272a4 } /* Comment.PreprocFile */
+	.c1 { color: #6272a4 } /* Comment.Single */
+	.cs { color: #6272a4 } /* Comment.Special */
+	.gd { color: #8b080b } /* Generic.Deleted */
+	.ge { color: #f8f8f2; text-decoration: underline } /* Generic.Emph */
+	.gr { color: #f8f8f2 } /* Generic.Error */
+	.gh { color: #f8f8f2; font-weight: bold } /* Generic.Heading */
+	.gi { color: #f8f8f2; font-weight: bold } /* Generic.Inserted */
+	.go { color: #44475a } /* Generic.Output */
+	.gp { color: #f8f8f2 } /* Generic.Prompt */
+	.gs { color: #f8f8f2 } /* Generic.Strong */
+	.gu { color: #f8f8f2; font-weight: bold } /* Generic.Subheading */
+	.gt { color: #f8f8f2 } /* Generic.Traceback */
+	.kc { color: #ff79c6 } /* Keyword.Constant */
+	.kd { color: #8be9fd; font-style: italic } /* Keyword.Declaration */
+	.kn { color: #ff79c6 } /* Keyword.Namespace */
+	.kp { color: #ff79c6 } /* Keyword.Pseudo */
+	.kr { color: #ff79c6 } /* Keyword.Reserved */
+	.kt { color: #8be9fd } /* Keyword.Type */
+	.ld { color: #f8f8f2 } /* Literal.Date */
+	.m { color: #bd93f9 } /* Literal.Number */
+	.s { color: #f1fa8c } /* Literal.String */
+	.na { color: #50fa7b } /* Name.Attribute */
+	.nb { color: #8be9fd; font-style: italic } /* Name.Builtin */
+	.nc { color: #50fa7b } /* Name.Class */
+	.no { color: #f8f8f2 } /* Name.Constant */
+	.nd { color: #f8f8f2 } /* Name.Decorator */
+	.ni { color: #f8f8f2 } /* Name.Entity */
+	.ne { color: #f8f8f2 } /* Name.Exception */
+	.nf { color: #50fa7b } /* Name.Function */
+	.nl { color: #8be9fd; font-style: italic } /* Name.Label */
+	.nn { color: #f8f8f2 } /* Name.Namespace */
+	.nx { color: #f8f8f2 } /* Name.Other */
+	.py { color: #f8f8f2 } /* Name.Property */
+	.nt { color: #ff79c6 } /* Name.Tag */
+	.nv { color: #8be9fd; font-style: italic } /* Name.Variable */
+	.ow { color: #ff79c6 } /* Operator.Word */
+	.w { color: #f8f8f2 } /* Text.Whitespace */
+	.mb { color: #bd93f9 } /* Literal.Number.Bin */
+	.mf { color: #bd93f9 } /* Literal.Number.Float */
+	.mh { color: #bd93f9 } /* Literal.Number.Hex */
+	.mi { color: #bd93f9 } /* Literal.Number.Integer */
+	.mo { color: #bd93f9 } /* Literal.Number.Oct */
+	.sa { color: #f1fa8c } /* Literal.String.Affix */
+	.sb { color: #f1fa8c } /* Literal.String.Backtick */
+	.sc { color: #f1fa8c } /* Literal.String.Char */
+	.dl { color: #f1fa8c } /* Literal.String.Delimiter */
+	.sd { color: #f1fa8c } /* Literal.String.Doc */
+	.s2 { color: #f1fa8c } /* Literal.String.Double */
+	.se { color: #f1fa8c } /* Literal.String.Escape */
+	.sh { color: #f1fa8c } /* Literal.String.Heredoc */
+	.si { color: #f1fa8c } /* Literal.String.Interpol */
+	.sx { color: #f1fa8c } /* Literal.String.Other */
+	.sr { color: #f1fa8c } /* Literal.String.Regex */
+	.s1 { color: #f1fa8c } /* Literal.String.Single */
+	.ss { color: #f1fa8c } /* Literal.String.Symbol */
+	.bp { color: #f8f8f2; font-style: italic } /* Name.Builtin.Pseudo */
+	.fm { color: #50fa7b } /* Name.Function.Magic */
+	.vc { color: #8be9fd; font-style: italic } /* Name.Variable.Class */
+	.vg { color: #8be9fd; font-style: italic } /* Name.Variable.Global */
+	.vi { color: #8be9fd; font-style: italic } /* Name.Variable.Instance */
+	.vm { color: #8be9fd; font-style: italic } /* Name.Variable.Magic */
+	.il { color: #bd93f9 } /* Literal.Number.Integer.Long */
+}

app/scss/packagegrid.scss

@@ -38,35 +38,43 @@ li.d-flex {
 	bottom: 0;
 	left: 0;
 	padding: 1em;
+
+	h3 {
+		color: white;
+		font-size: 120%;
+		font-weight: bold;
+		margin: 0;
+		padding: 0;
+	}
+
+	small {
+		color: #ddd;
+		font-size: 75%;
+		font-weight: bold;
+	}
+
+	p {
+		display: none;
+		color: #ddd;
+		font-weight: normal;
+	}
 }
 
-.packagegridinfo h3 {
-	color: white;
-	font-size: 120%;
-	font-weight: bold;
-}
+.packagetile a:hover  {
+	.packagegridinfo {
+		top: 0;
+	}
 
-.packagegridinfo small {
-	color: #ddd;
-	font-size: 75%;
-	font-weight: bold;
-}
+	h3 {
+		margin-bottom: 0.5em;
+	}
 
-.packagegridinfo p {
-	display: none;
-	color: #ddd;
-	font-weight: normal;
-}
+	p {
+		display: block;
+	}
 
-.packagetile a:hover .packagegridinfo {
-	top: 0;
-}
-
-.packagetile a:hover p {
-	display: block;
-}
-
-.packagetile a:hover .packagegridscrub {
-	top: 0;
-	background: rgba(0,0,0,0.8);
+	.packagegridscrub {
+		top: 0;
+		background: rgba(0,0,0,0.8);
+	}
 }

app/scss/packages.scss

@@ -53,6 +53,24 @@
 		color: rgba(255, 255, 255, 0.8);
 	}
 
+	.btn-group-horizontal > span {
+		color: rgba(255, 255, 255, 0.8);
+		display: inline-block;
+		font-weight: 400;
+		text-align: center;
+		white-space: nowrap;
+		vertical-align: middle;
+		-webkit-user-select: none;
+		-moz-user-select: none;
+		-ms-user-select: none;
+		user-select: none;
+		border: 1px solid transparent;
+		padding: 0.375rem 0.75rem;
+		font-size: 0.9375rem;
+		line-height: 1.5;
+		border-radius: 0.25rem;
+	}
+
 	a:hover {
 		color: #fff;
 		text-decoration: none;

app/tasks/__init__.py

@@ -1,26 +1,25 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
+from logging import Filter
 
 import flask
-from flask_sqlalchemy import SQLAlchemy
-from celery import Celery
+from celery import Celery, signals
 from celery.schedules import crontab
 from app import app
-from app.models import *
+
 
 class TaskError(Exception):
 	def __init__(self, value):
@@ -37,18 +36,18 @@ class FlaskCelery(Celery):
 			self.init_app(kwargs['app'])
 
 	def patch_task(self):
-		TaskBase = self.Task
+		BaseTask : celery.Task = self.Task
 		_celery = self
 
-		class ContextTask(TaskBase):
+		class ContextTask(BaseTask):
 			abstract = True
 
 			def __call__(self, *args, **kwargs):
 				if flask.has_app_context():
-					return TaskBase.__call__(self, *args, **kwargs)
+					return super(BaseTask, self).__call__(*args, **kwargs)
 				else:
 					with _celery.app.app_context():
-						return TaskBase.__call__(self, *args, **kwargs)
+						return super(BaseTask, self).__call__(*args, **kwargs)
 
 		self.Task = ContextTask
 
@@ -73,8 +72,40 @@ CELERYBEAT_SCHEDULE = {
 	'package_score_update': {
 		'task': 'app.tasks.pkgtasks.updatePackageScores',
 		'schedule': crontab(minute=10, hour=1),
+	},
+	'check_for_updates': {
+		'task': 'app.tasks.importtasks.check_for_updates',
+		'schedule': crontab(minute=10, hour=1),
+	},
+	'send_pending_notifications': {
+		'task': 'app.tasks.emails.send_pending_notifications',
+		'schedule': crontab(minute='*/5'),
+	},
+	'send_notification_digests': {
+		'task': 'app.tasks.emails.send_pending_digests',
+		'schedule': crontab(minute=0, hour=14),
 	}
 }
 celery.conf.beat_schedule = CELERYBEAT_SCHEDULE
 
-from . import importtasks, forumtasks, emails, pkgtasks
+from . import importtasks, forumtasks, emails, pkgtasks, celery
+
+
+# noinspection PyUnusedLocal
+@signals.after_setup_logger.connect
+def on_after_setup_logger(**kwargs):
+	from app.maillogger import build_handler
+
+	class ExceptionFilter(Filter):
+		def filter(self, record):
+			if record.exc_info:
+				exc, _, _ = record.exc_info
+				if exc == TaskError:
+					return False
+
+			return True
+
+	logger = celery.log.get_default_logger()
+	handler = build_handler(app)
+	handler.addFilter(ExceptionFilter())
+	logger.addHandler(handler)

app/tasks/emails.py

@@ -1,30 +1,47 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-from flask import render_template, url_for
+from flask import render_template
 from flask_mail import Message
 from app import mail
+from app.models import Notification, db, EmailSubscription, User
 from app.tasks import celery
-from app.utils import abs_url_for
+from app.utils import abs_url_for, abs_url, randomString
+
+
+def get_email_subscription(email):
+	assert type(email) == str
+	ret = EmailSubscription.query.filter_by(email=email).first()
+	if not ret:
+		ret = EmailSubscription(email)
+		ret.token = randomString(32)
+		db.session.add(ret)
+		db.session.commit()
+
+	return ret
+
 
 @celery.task()
-def sendVerifyEmail(newEmail, token):
-	print("Sending verify email!")
-	msg = Message("Verify email address", recipients=[newEmail])
+def send_verify_email(email, token):
+	sub = get_email_subscription(email)
+	if sub.blacklisted:
+		return
+
+	msg = Message("Confirm email address", recipients=[email])
 
 	msg.body = """
 			This email has been sent to you because someone (hopefully you)
@@ -32,20 +49,131 @@ def sendVerifyEmail(newEmail, token):
 
 			If it wasn't you, then just delete this email.
 
-			If this was you, then please click this link to verify the address:
+			If this was you, then please click this link to confirm the address:
 
 			{}
 		""".format(abs_url_for('users.verify_email', token=token))
 
-	msg.html = render_template("emails/verify.html", token=token)
+	msg.html = render_template("emails/verify.html", token=token, sub=sub)
 	mail.send(msg)
 
+
 @celery.task()
-def sendEmailRaw(to, subject, text, html):
-	from flask_mail import Message
-	msg = Message(subject, recipients=to)
+def send_unsubscribe_verify(email):
+	sub = get_email_subscription(email)
+	if sub.blacklisted:
+		return
 
-	msg.body = text or html
-	html = html or text
-	msg.html = render_template("emails/base.html", subject=subject, content=html)
+	msg = Message("Confirm unsubscribe", recipients=[email])
+
+	msg.body = """
+				We're sorry to see you go. You just need to do one more thing before your email is blacklisted.
+				
+				Click this link to blacklist email: {} 
+			""".format(abs_url_for('users.unsubscribe', token=sub.token))
+
+	msg.html = render_template("emails/verify_unsubscribe.html", sub=sub)
 	mail.send(msg)
+
+
+@celery.task()
+def send_email_with_reason(email, subject, text, html, reason):
+	sub = get_email_subscription(email)
+	if sub.blacklisted:
+		return
+
+	from flask_mail import Message
+	msg = Message(subject, recipients=[email])
+
+	msg.body = text
+	html = html or text
+	msg.html = render_template("emails/base.html", subject=subject, content=html, reason=reason, sub=sub)
+	mail.send(msg)
+
+
+@celery.task()
+def send_user_email(email: str, subject: str, text: str, html=None):
+	return send_email_with_reason(email, subject, text, html,
+			"You are receiving this email because you are a registered user of ContentDB.")
+
+
+@celery.task()
+def send_anon_email(email: str, subject: str, text: str, html=None):
+	return send_email_with_reason(email, subject, text, html,
+			"You are receiving this email because someone (hopefully you) entered your email address as a user's email.")
+
+
+def send_single_email(notification):
+	sub = get_email_subscription(notification.user.email)
+	if sub.blacklisted:
+		return
+
+	msg = Message(notification.title, recipients=[notification.user.email])
+
+	msg.body = """
+			New notification: {}
+			
+			View: {}
+			
+			Manage email settings: {}
+			Unsubscribe: {}
+		""".format(notification.title, abs_url(notification.url),
+					abs_url_for("users.email_notifications", username=notification.user.username),
+					abs_url_for("users.unsubscribe", token=sub.token))
+
+	msg.html = render_template("emails/notification.html", notification=notification, sub=sub)
+	mail.send(msg)
+
+
+def send_notification_digest(notifications: [Notification]):
+	user = notifications[0].user
+
+	sub = get_email_subscription(user.email)
+	if sub.blacklisted:
+		return
+
+	msg = Message("{} new notifications".format(len(notifications)), recipients=[user.email])
+
+	msg.body = "".join(["<{}> {}\nView: {}\n\n".format(notification.causer.display_name, notification.title, abs_url(notification.url)) for notification in notifications])
+
+	msg.body += "Manage email settings: {}\nUnsubscribe: {}".format(
+			abs_url_for("users.email_notifications", username=user.username),
+			abs_url_for("users.unsubscribe", token=sub.token))
+
+	msg.html = render_template("emails/notification_digest.html", notifications=notifications, user=user, sub=sub)
+	mail.send(msg)
+
+
+@celery.task()
+def send_pending_digests():
+	for user in User.query.filter(User.notifications.any(emailed=False)).all():
+		to_send = []
+		for notification in user.notifications:
+			if not notification.emailed and notification.can_send_digest():
+				to_send.append(notification)
+				notification.emailed = True
+
+		if len(to_send) > 0:
+			send_notification_digest(to_send)
+
+		db.session.commit()
+
+
+@celery.task()
+def send_pending_notifications():
+	for user in User.query.filter(User.notifications.any(emailed=False)).all():
+		to_send = []
+		for notification in user.notifications:
+			if not notification.emailed:
+				if notification.can_send_email():
+					to_send.append(notification)
+					notification.emailed = True
+				elif not notification.can_send_digest():
+					notification.emailed = True
+
+		db.session.commit()
+
+		if len(to_send) > 1:
+			send_notification_digest(to_send)
+		elif len(to_send) > 0:
+			send_single_email(to_send[0])

app/tasks/forumtasks.py

@@ -1,28 +1,25 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-import flask, json, re
-from flask_sqlalchemy import SQLAlchemy
-from app import app
+import json, re, sys
 from app.models import *
 from app.tasks import celery
-from .phpbbparser import getProfile, getTopicsFromForum
+from app.utils.phpbbparser import getProfile, getTopicsFromForum
 import urllib.request
-from urllib.parse import urlparse, quote_plus
 
 @celery.task()
 def checkForumAccount(username, forceNoSave=False):
@@ -32,6 +29,9 @@ def checkForumAccount(username, forceNoSave=False):
 	except OSError:
 		return
 
+	if profile is None:
+		return
+
 	user = User.query.filter_by(forums_username=username).first()
 
 	# Create user
@@ -64,6 +64,7 @@ def checkForumAccount(username, forceNoSave=False):
 
 	return needsSaving
 
+
 @celery.task()
 def checkAllForumAccounts(forceNoSave=False):
 	needsSaving = False
@@ -131,21 +132,34 @@ def importTopicList():
 	for topic in ForumTopic.query.all():
 		topics_by_id[topic.topic_id] = topic
 
+	def get_or_create_user(username):
+		user = username_to_user.get(username)
+		if user:
+			return user
+
+		user = User.query.filter_by(forums_username=username).first()
+		if user is None:
+			user = User.query.filter_by(username=username).first()
+			if user:
+				return None
+
+			user = User(username)
+			user.forums_username = username
+			db.session.add(user)
+
+		username_to_user[username] = user
+		return user
+
 	# Create or update
 	for info in info_by_id.values():
 		id = int(info["id"])
 
 		# Get author
 		username = info["author"]
-		user = username_to_user.get(username)
+		user = get_or_create_user(username)
 		if user is None:
-			user = User.query.filter_by(forums_username=username).first()
-			if user is None:
-				print(username + " not found!")
-				user = User(username)
-				user.forums_username = username
-				db.session.add(user)
-			username_to_user[username] = user
+			print("Error! Unable to create user {}".format(username), file=sys.stderr)
+			continue
 
 		# Get / add row
 		topic = topics_by_id.get(id)

app/tasks/importtasks.py

@@ -1,239 +1,125 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21  rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
-
-
-import flask, json, os, git, tempfile, shutil, gitdb
+import json
+import os, shutil, gitdb
+from zipfile import ZipFile
 from git import GitCommandError
 from git_archive_all import GitArchiver
-from flask_sqlalchemy import SQLAlchemy
-from urllib.error import HTTPError
-import urllib.request
-from urllib.parse import urlparse, quote_plus, urlsplit
-from zipfile import ZipFile
+from kombu import uuid
 
-from app import app
 from app.models import *
 from app.tasks import celery, TaskError
-from app.utils import randomString
+from app.utils import randomString, post_bot_message, addSystemNotification, addSystemAuditLog, get_system_user
+from app.utils.git import clone_repo, get_latest_tag, get_latest_commit, get_temp_dir
 from .minetestcheck import build_tree, MinetestCheckError, ContentType
-from .minetestcheck.config import parse_conf
+from ..logic.LogicError import LogicError
+from ..logic.packages import do_edit_package
 
-class GithubURLMaker:
-	def __init__(self, url):
-		self.baseUrl = None
-		self.user = None
-		self.repo = None
-
-		# Rewrite path
-		import re
-		m = re.search("^\/([^\/]+)\/([^\/]+)\/?$", url.path)
-		if m is None:
-			return
-
-		user = m.group(1)
-		repo = m.group(2).replace(".git", "")
-		self.baseUrl = "https://raw.githubusercontent.com/{}/{}/master" \
-				.format(user, repo)
-		self.user = user
-		self.repo = repo
-
-	def isValid(self):
-		return self.baseUrl is not None
-
-	def getRepoURL(self):
-		return "https://github.com/{}/{}".format(self.user, self.repo)
-
-	def getScreenshotURL(self):
-		return self.baseUrl + "/screenshot.png"
-
-	def getModConfURL(self):
-		return self.baseUrl + "/mod.conf"
-
-	def getCommitsURL(self, branch):
-		return "https://api.github.com/repos/{}/{}/commits?sha={}" \
-				.format(self.user, self.repo, urllib.parse.quote_plus(branch))
-
-	def getCommitDownload(self, commit):
-		return "https://github.com/{}/{}/archive/{}.zip" \
-				.format(self.user, self.repo, commit)
-
-krock_list_cache = None
-krock_list_cache_by_name = None
-def getKrockList():
-	global krock_list_cache
-	global krock_list_cache_by_name
-
-	if krock_list_cache is None:
-		contents = urllib.request.urlopen("https://krock-works.uk.to/minetest/modList.php").read().decode("utf-8")
-		list = json.loads(contents)
-
-		def h(x):
-			if not ("title"   in x and "author" in x and \
-					"topicId" in x and "link"   in x and x["link"] != ""):
-				return False
-
-			import re
-			m = re.search("\[([A-Za-z0-9_]+)\]", x["title"])
-			if m is None:
-				return False
-
-			x["name"] = m.group(1)
-			return True
-
-		def g(x):
-			return {
-				"title":   x["title"],
-				"author":  x["author"],
-				"name":	x["name"],
-				"topicId": x["topicId"],
-				"link":	x["link"],
-			}
-
-		krock_list_cache = [g(x) for x in list if h(x)]
-		krock_list_cache_by_name = {}
-		for x in krock_list_cache:
-			if not x["name"] in krock_list_cache_by_name:
-				krock_list_cache_by_name[x["name"]] = []
-
-			krock_list_cache_by_name[x["name"]].append(x)
-
-	return krock_list_cache, krock_list_cache_by_name
-
-def findModInfo(author, name, link):
-	list, lookup = getKrockList()
-
-	if name is not None and name in lookup:
-		if len(lookup[name]) == 1:
-			return lookup[name][0]
-
-		for x in lookup[name]:
-			if x["author"] == author:
-				return x
-
-	if link is not None and len(link) > 15:
-		for x in list:
-			if link in x["link"]:
-				return x
-
-	return None
-
-def generateGitURL(urlstr):
-	scheme, netloc, path, query, frag = urlsplit(urlstr)
-
-	return "http://:@" + netloc + path + query
-
-
-def getTempDir():
-	return os.path.join(tempfile.gettempdir(), randomString(10))
-
-
-# Clones a repo from an unvalidated URL.
-# Returns a tuple of path and repo on sucess.
-# Throws `TaskError` on failure.
-# Caller is responsible for deleting returned directory.
-def cloneRepo(urlstr, ref=None, recursive=False):
-	gitDir = getTempDir()
-
-	err = None
-	try:
-		gitUrl = generateGitURL(urlstr)
-		print("Cloning from " + gitUrl)
-
-		if ref is None:
-			repo = git.Repo.clone_from(gitUrl, gitDir, \
-					progress=None, env=None, depth=1, recursive=recursive, kill_after_timeout=15)
-		else:
-			repo = git.Repo.init(gitDir)
-			origin = repo.create_remote("origin", url=gitUrl)
-			assert origin.exists()
-			origin.fetch()
-			origin.pull(ref)
-
-			for submodule in repo.submodules:
-				submodule.update(init=True)
-
-		return gitDir, repo
-
-	except GitCommandError as e:
-		# This is needed to stop the backtrace being weird
-		err = e.stderr
-
-	except gitdb.exc.BadName as e:
-		err = "Unable to find the reference " + (ref or "?") + "\n" + e.stderr
-
-	raise TaskError(err.replace("stderr: ", "") \
-			.replace("Cloning into '" + gitDir + "'...", "") \
-			.strip())
 
 @celery.task()
 def getMeta(urlstr, author):
-	gitDir, _ = cloneRepo(urlstr, recursive=True)
+	with clone_repo(urlstr, recursive=True) as repo:
+		try:
+			tree = build_tree(repo.working_tree_dir, author=author, repo=urlstr)
+		except MinetestCheckError as err:
+			raise TaskError(str(err))
 
+		result = {"name": tree.name, "provides": tree.getModNames(), "type": tree.type.name}
+
+		for key in ["depends", "optional_depends"]:
+			result[key] = tree.fold("meta", key)
+
+		for key in ["title", "repo", "issueTracker", "forumId", "description", "short_description"]:
+			result[key] = tree.get(key)
+
+		for mod in result["provides"]:
+			result["depends"].discard(mod)
+			result["optional_depends"].discard(mod)
+
+		for key, value in result.items():
+			if isinstance(value, set):
+				result[key] = list(value)
+
+		return result
+
+
+def postReleaseCheckUpdate(self, release, path):
 	try:
-		tree = build_tree(gitDir, author=author, repo=urlstr)
+		tree = build_tree(path, expected_type=ContentType[release.package.type.name],
+				author=release.package.author.username, name=release.package.name)
+
+		cache = {}
+		def getMetaPackages(names):
+			return [ MetaPackage.GetOrCreate(x, cache) for x in names ]
+
+		provides = tree.getModNames()
+
+		package = release.package
+		package.provides.clear()
+		package.provides.extend(getMetaPackages(tree.getModNames()))
+
+		# Delete all meta package dependencies
+		package.dependencies.filter(Dependency.meta_package != None).delete()
+
+		# Get raw dependencies
+		depends = tree.fold("meta", "depends")
+		optional_depends = tree.fold("meta", "optional_depends")
+
+		# Filter out provides
+		for mod in provides:
+			depends.discard(mod)
+			optional_depends.discard(mod)
+
+		# Add dependencies
+		for meta in getMetaPackages(depends):
+			db.session.add(Dependency(package, meta=meta, optional=False))
+
+		for meta in getMetaPackages(optional_depends):
+			db.session.add(Dependency(package, meta=meta, optional=True))
+
+		# Update min/max
+		if tree.meta.get("min_minetest_version"):
+			release.min_rel = MinetestRelease.get(tree.meta["min_minetest_version"], None)
+
+		if tree.meta.get("max_minetest_version"):
+			release.max_rel = MinetestRelease.get(tree.meta["max_minetest_version"], None)
+
+		try:
+			with open(os.path.join(tree.baseDir, ".cdb.json"), "r") as f:
+				data = json.loads(f.read())
+				do_edit_package(package.author, package, False, data, "Post release hook")
+		except LogicError as e:
+			raise TaskError(e.message)
+		except IOError:
+			pass
+
+		return tree
+
 	except MinetestCheckError as err:
+		db.session.rollback()
+
+		if "Fails validation" not in release.title:
+			release.title += " (Fails validation)"
+
+		release.task_id = self.request.id
+		release.approved = False
+		db.session.commit()
+
 		raise TaskError(str(err))
 
-	shutil.rmtree(gitDir)
-
-	result = {}
-	result["name"] = tree.name
-	result["provides"] = tree.fold("name")
-	result["type"] = tree.type.name
-
-	for key in ["depends", "optional_depends"]:
-		result[key] = tree.fold("meta", key)
-
-	for key in ["title", "repo", "issueTracker", "forumId", "description", "short_description"]:
-		result[key] = tree.get(key)
-
-	for mod in result["provides"]:
-		result["depends"].discard(mod)
-		result["optional_depends"].discard(mod)
-
-	for key, value in result.items():
-		if isinstance(value, set):
-			result[key] = list(value)
-
-	return result
-
-
-def makeVCSReleaseFromGithub(id, branch, release, url):
-	urlmaker = GithubURLMaker(url)
-	if not urlmaker.isValid():
-		raise TaskError("Invalid github repo URL")
-
-	commitsURL = urlmaker.getCommitsURL(branch)
-	try:
-		contents = urllib.request.urlopen(commitsURL).read().decode("utf-8")
-		commits = json.loads(contents)
-	except HTTPError:
-		raise TaskError("Unable to get commits for Github repository. Either the repository or reference doesn't exist.")
-
-	if len(commits) == 0 or not "sha" in commits[0]:
-		raise TaskError("No commits found")
-
-	release.url          = urlmaker.getCommitDownload(commits[0]["sha"])
-	release.task_id     = None
-	release.commit_hash = commits[0]["sha"]
-	release.approve(release.package.author)
-	db.session.commit()
-
-	return release.url
-
 
 @celery.task(bind=True)
 def checkZipRelease(self, id, path):
@@ -243,58 +129,33 @@ def checkZipRelease(self, id, path):
 	elif release.package is None:
 		raise TaskError("No package attached to release")
 
-	temp = getTempDir()
-	try:
+	with get_temp_dir() as temp:
 		with ZipFile(path, 'r') as zip_ref:
 			zip_ref.extractall(temp)
 
-		try:
-			tree = build_tree(temp, expected_type=ContentType[release.package.type.name], \
-				author=release.package.author.username, name=release.package.name)
-		except MinetestCheckError as err:
-			if "Fails validation" not in release.title:
-				release.title += " (Fails validation)"
-
-			release.task_id = self.request.id
-			release.approved = False
-			db.session.commit()
-
-			raise TaskError(str(err))
+		postReleaseCheckUpdate(self, release, temp)
 
 		release.task_id = None
 		release.approve(release.package.author)
 		db.session.commit()
 
-	finally:
-		shutil.rmtree(temp)
 
-
-@celery.task()
-def makeVCSRelease(id, branch):
+@celery.task(bind=True)
+def makeVCSRelease(self, id, branch):
 	release = PackageRelease.query.get(id)
 	if release is None:
 		raise TaskError("No such release!")
 	elif release.package is None:
 		raise TaskError("No package attached to release")
 
-	# url = urlparse(release.package.repo)
-	# if url.netloc == "github.com":
-	# 	return makeVCSReleaseFromGithub(id, branch, release, url)
+	with clone_repo(release.package.repo, ref=branch, recursive=True) as repo:
+		postReleaseCheckUpdate(self, release, repo.working_tree_dir)
 
-	gitDir, repo = cloneRepo(release.package.repo, ref=branch, recursive=True)
-
-	try:
-		tree = build_tree(gitDir, expected_type=ContentType[release.package.type.name], \
-			author=release.package.author.username, name=release.package.name)
-	except MinetestCheckError as err:
-		raise TaskError(str(err))
-
-	try:
 		filename = randomString(10) + ".zip"
 		destPath = os.path.join(app.config["UPLOAD_DIR"], filename)
 
 		assert(not os.path.isfile(destPath))
-		archiver = GitArchiver(force_sub=True, main_repo_abspath=gitDir)
+		archiver = GitArchiver(prefix=release.package.name, force_sub=True, main_repo_abspath=repo.working_tree_dir)
 		archiver.create(destPath)
 		assert(os.path.isfile(destPath))
 
@@ -302,136 +163,161 @@ def makeVCSRelease(id, branch):
 		release.task_id     = None
 		release.commit_hash = repo.head.object.hexsha
 		release.approve(release.package.author)
-		print(release.url)
 		db.session.commit()
 
 		return release.url
-	finally:
-		shutil.rmtree(gitDir)
+
 
 @celery.task()
 def importRepoScreenshot(id):
 	package = Package.query.get(id)
-	if package is None or package.soft_deleted:
+	if package is None or package.state == PackageState.DELETED:
 		raise Exception("Unexpected none package")
 
-	# Get URL Maker
 	try:
-		gitDir, _ = cloneRepo(package.repo)
+		with clone_repo(package.repo) as repo:
+			for ext in ["png", "jpg", "jpeg"]:
+				sourcePath = repo.working_tree_dir + "/screenshot." + ext
+				if os.path.isfile(sourcePath):
+					filename = randomString(10) + "." + ext
+					destPath = os.path.join(app.config["UPLOAD_DIR"], filename)
+					shutil.copyfile(sourcePath, destPath)
+
+					ss = PackageScreenshot()
+					ss.approved = True
+					ss.package = package
+					ss.title   = "screenshot.png"
+					ss.url	 = "/uploads/" + filename
+					db.session.add(ss)
+					db.session.commit()
+
+					return "/uploads/" + filename
+
 	except TaskError as e:
 		# ignore download errors
 		print(e)
-		return None
-
-	# Find and import screenshot
-	try:
-		for ext in ["png", "jpg", "jpeg"]:
-			sourcePath = gitDir + "/screenshot." + ext
-			if os.path.isfile(sourcePath):
-				filename = randomString(10) + "." + ext
-				destPath = os.path.join(app.config["UPLOAD_DIR"], filename)
-				shutil.copyfile(sourcePath, destPath)
-
-				ss = PackageScreenshot()
-				ss.approved = True
-				ss.package = package
-				ss.title   = "screenshot.png"
-				ss.url	 = "/uploads/" + filename
-				db.session.add(ss)
-				db.session.commit()
-
-				return "/uploads/" + filename
-	finally:
-		shutil.rmtree(gitDir)
+		pass
 
 	print("screenshot.png does not exist")
 	return None
 
 
+def check_update_config_impl(package):
+	config = package.update_config
 
-def getDepends(package):
-	url = urlparse(package.repo)
-	urlmaker = None
-	if url.netloc == "github.com":
-		urlmaker = GithubURLMaker(url)
+	if config.trigger == PackageUpdateTrigger.COMMIT:
+		tag = None
+		commit = get_latest_commit(package.repo, package.update_config.ref)
+	elif config.trigger == PackageUpdateTrigger.TAG:
+		tag, commit = get_latest_tag(package.repo)
 	else:
-		return {}
+		raise TaskError("Unknown update trigger")
 
-	result = {}
-	if not urlmaker.isValid():
-		return {}
-
-	#
-	# Try getting depends on mod.conf
-	#
-	try:
-		contents = urllib.request.urlopen(urlmaker.getModConfURL()).read().decode("utf-8")
-		conf = parse_conf(contents)
-		for key in ["depends", "optional_depends"]:
-			try:
-				result[key] = conf[key]
-			except KeyError:
-				pass
-
-	except HTTPError:
-		print("mod.conf does not exist")
-
-	if "depends" in result or "optional_depends" in result:
-		return result
-
-
-	#
-	# Try depends.txt
-	#
-	import re
-	pattern = re.compile("^([a-z0-9_]+)\??$")
-	try:
-		contents = urllib.request.urlopen(urlmaker.getDependsURL()).read().decode("utf-8")
-		soft = []
-		hard = []
-		for line in contents.split("\n"):
-			line = line.strip()
-			if pattern.match(line):
-				if line[len(line) - 1] == "?":
-					soft.append( line[:-1])
-				else:
-					hard.append(line)
-
-		result["depends"] = ",".join(hard)
-		result["optional_depends"] = ",".join(soft)
-	except HTTPError:
-		print("depends.txt does not exist")
-
-	return result
-
-
-def importDependencies(package, mpackage_cache):
-	if Dependency.query.filter_by(depender=package).count() != 0:
+	if commit is None:
 		return
 
-	result = getDepends(package)
+	if config.last_commit == commit:
+		if tag and config.last_tag != tag:
+			config.last_tag = tag
+			db.session.commit()
+		return
 
-	if "depends" in result:
-		deps = Dependency.SpecToList(package, result["depends"], mpackage_cache)
-		print("{} hard: {}".format(len(deps), result["depends"]))
-		for dep in deps:
-			dep.optional = False
-			db.session.add(dep)
+	if not config.last_commit:
+		config.last_commit = commit
+		config.last_tag = tag
+		db.session.commit()
+		return
 
-	if "optional_depends" in result:
-		deps = Dependency.SpecToList(package, result["optional_depends"], mpackage_cache)
-		print("{} soft: {}".format(len(deps), result["optional_depends"]))
-		for dep in deps:
-			dep.optional = True
-			db.session.add(dep)
+	if package.releases.filter_by(commit_hash=commit).count() > 0:
+		return
 
-@celery.task()
-def importAllDependencies():
-	Dependency.query.delete()
-	mpackage_cache = {}
-	packages = Package.query.filter_by(type=PackageType.MOD).all()
-	for i, p in enumerate(packages):
-		print("============= {} ({}/{}) =============".format(p.name, i, len(packages)))
-		importDependencies(p, mpackage_cache)
+	if config.make_release:
+		rel = PackageRelease()
+		rel.package = package
+		rel.title = tag if tag else datetime.datetime.utcnow().strftime("%Y-%m-%d")
+		rel.url = ""
+		rel.task_id = uuid()
+		db.session.add(rel)
+
+		msg = "Created release {} (Git Update Detection)".format(rel.title)
+		addSystemAuditLog(AuditSeverity.NORMAL, msg, package.getDetailsURL(), package)
+
+		db.session.commit()
+
+		makeVCSRelease.apply_async((rel.id, commit), task_id=rel.task_id)
+
+	elif config.outdated_at is None:
+		config.set_outdated()
+
+		if config.trigger == PackageUpdateTrigger.COMMIT:
+			msg_last = ""
+			if config.last_commit:
+				msg_last = " The last commit was {}".format(config.last_commit[0:5])
+
+			msg = "New commit {} found on the Git repo, is the package outdated?{}" \
+				.format(commit[0:5], msg_last)
+		else:
+			msg_last = ""
+			if config.last_tag:
+				msg_last = " The last tag was {}".format(config.last_tag)
+
+			msg = "New tag {} found on the Git repo.{}" \
+				.format(tag, msg_last)
+
+		for user in package.maintainers:
+			addSystemNotification(user, NotificationType.BOT,
+					msg, url_for("todo.view_user", username=user.username, _external=False), package)
+
+	config.last_commit = commit
+	config.last_tag = tag
+	db.session.commit()
+
+
+@celery.task(bind=True)
+def check_update_config(self, package_id):
+	package: Package = Package.query.get(package_id)
+	if package is None:
+		raise TaskError("No such package!")
+	elif package.update_config is None:
+		raise TaskError("No update config attached to package")
+
+	err = None
+	try:
+		check_update_config_impl(package)
+	except GitCommandError as e:
+		# This is needed to stop the backtrace being weird
+		err = e.stderr
+	except gitdb.exc.BadName as e:
+		err = "Unable to find the reference " + (package.update_config.ref or "?") + "\n" + e.stderr
+	except TaskError as e:
+		err = e.value
+
+	if err:
+		err = err.replace("stderr: ", "") \
+			.replace("Cloning into '/tmp/", "Cloning into '") \
+			.strip()
+
+		msg = "Error: {}.\n\nTask ID: {}\n\n[Change update configuration]({})" \
+			.format(err, self.request.id, package.getUpdateConfigURL())
+
+		post_bot_message(package, "Failed to check git repository", msg)
+
+		db.session.commit()
+		return
+
+
+@celery.task
+def check_for_updates():
+	for update_config in PackageUpdateConfig.query.all():
+		update_config: PackageUpdateConfig
+
+		if not update_config.package.approved:
+			continue
+
+		if update_config.package.repo is None:
+			db.session.delete(update_config)
+			continue
+
+		check_update_config.delay(update_config.package_id)
 
 	db.session.commit()

app/tasks/minetestcheck/__init__.py

@@ -20,18 +20,18 @@ class ContentType(Enum):
 		"""
 		Whether or not `other` is an acceptable type for this
 		"""
-		assert(other)
+		assert other
 
 		if self == ContentType.MOD:
 			if not other.isModLike():
-				raise MinetestCheckError("expected a mod or modpack, found " + other.value)
+				raise MinetestCheckError("Expected a mod or modpack, found " + other.value)
 
 		elif self == ContentType.TXP:
 			if other != ContentType.UNKNOWN and other != ContentType.TXP:
 				raise MinetestCheckError("expected a " + self.value + ", found a " + other.value)
 
 		elif other != self:
-			raise MinetestCheckError("expected a " + self.value + ", found a " + other.value)
+			raise MinetestCheckError("Expected a " + self.value + ", found a " + other.value)
 
 
 from .tree import PackageTreeNode, get_base_dir
@@ -40,7 +40,7 @@ def build_tree(path, expected_type=None, author=None, repo=None, name=None):
 	path = get_base_dir(path)
 
 	root = PackageTreeNode(path, "/", author=author, repo=repo, name=name)
-	assert(root)
+	assert root
 
 	if expected_type:
 		expected_type.validate_same(root.type)

app/tasks/minetestcheck/config.py

@@ -1,10 +1,56 @@
 def parse_conf(string):
 	retval = {}
-	for line in string.split("\n"):
-		idx = line.find("=")
-		if idx > 0:
-			key   = line[:idx].strip()
-			value = line[idx+1:].strip()
-			retval[key] = value
+	lines = string.splitlines()
+	i = 0
+
+	def syntax_error(message):
+		raise SyntaxError("Line {}: {}".format(i + 1, message))
+
+	while i < len(lines):
+		line = lines[i].strip()
+
+		# Comments
+		if line.startswith("#") or line == "":
+			i += 1
+			continue
+
+		key_value = line.split("=", 2)
+		if len(key_value) < 2:
+			syntax_error("Expected line to contain '='")
+
+		key = key_value[0].strip()
+		if key == "":
+			syntax_error("Missing key before '='")
+
+		value = key_value[1].strip()
+		if value.startswith('"""'):
+			value_lines = []
+			closed = False
+			i += 1
+			while i < len(lines):
+				value = lines[i]
+				if value == '"""':
+					closed = True
+					value = value[:-3]
+					break
+
+				value_lines.append(value)
+				i += 1
+
+			if not closed:
+				i -= 1
+				syntax_error("Unclosed multiline value")
+
+			value_lines.append(value)
+			value = "\n".join(value_lines)
+
+		else:
+			value = value.rstrip()
+
+		if key in retval:
+			syntax_error("Duplicate key {}".format(key))
+
+		retval[key] = value
+		i += 1
 
 	return retval

app/tasks/minetestcheck/tree.py

@@ -1,7 +1,9 @@
-import os
+import os, re
 from . import MinetestCheckError, ContentType
 from .config import parse_conf
 
+basenamePattern = re.compile("^([a-z0-9_]+)$")
+
 def get_base_dir(path):
 	if not os.path.isdir(path):
 		raise IOError("Expected dir")
@@ -29,9 +31,15 @@ def detect_type(path):
 		return ContentType.UNKNOWN
 
 
+def get_csv_line(line):
+	if line is None:
+		return []
+
+	return [x.strip() for x in line.split(",") if x.strip() != ""]
+
+
 class PackageTreeNode:
 	def __init__(self, baseDir, relative, author=None, repo=None, name=None):
-		print(baseDir)
 		self.baseDir  = baseDir
 		self.relative = relative
 		self.author   = author
@@ -46,26 +54,50 @@ class PackageTreeNode:
 
 		if self.type == ContentType.GAME:
 			if not os.path.isdir(baseDir + "/mods"):
-				raise MinetestCheckError(("game at {} does not have a mods/ folder").format(self.relative))
-			self.add_children_from_mod_dir(baseDir + "/mods")
+				raise MinetestCheckError("Game at {} does not have a mods/ folder".format(self.relative))
+			self.add_children_from_mod_dir("mods")
+		elif self.type == ContentType.MOD:
+			if self.name and not basenamePattern.match(self.name):
+				raise MinetestCheckError("Invalid base name for mod {} at {}, names must only contain a-z0-9_." \
+										 .format(self.name, self.relative))
 		elif self.type == ContentType.MODPACK:
-			self.add_children_from_mod_dir(baseDir)
+			self.add_children_from_mod_dir(None)
+
+
+	def getMetaFileName(self):
+		if self.type == ContentType.GAME:
+			return "game.conf"
+		elif self.type == ContentType.MOD:
+			return "mod.conf"
+		elif self.type == ContentType.MODPACK:
+			return "modpack.conf"
+		elif self.type == ContentType.TXP:
+			return "texture_pack.conf"
+		else:
+			return None
 
 
 	def read_meta(self):
 		result = {}
 
-		# .conf file
-		try:
-			with open(self.baseDir + "/mod.conf", "r") as myfile:
-				conf = parse_conf(myfile.read())
-				for key in ["name", "description", "title", "depends", "optional_depends"]:
-					try:
-						result[key] = conf[key]
-					except KeyError:
-						pass
-		except IOError:
-			pass
+		# Read .conf file
+		meta_file_name = self.getMetaFileName()
+		if meta_file_name is not None:
+			meta_file_rel = self.relative + meta_file_name
+			meta_file_path = self.baseDir + "/" + meta_file_name
+			try:
+				with open(meta_file_path or "", "r") as myfile:
+					conf = parse_conf(myfile.read())
+					for key, value in conf.items():
+						result[key] = value
+			except SyntaxError as e:
+				raise MinetestCheckError("Error while reading {}: {}".format(meta_file_rel , e.msg))
+			except IOError:
+				pass
+
+			if "release" in result:
+				raise MinetestCheckError("{} should not contain 'release' key, as this is for use by ContentDB only.".format(meta_file_rel))
+
 
 		# description.txt
 		if not "description" in result:
@@ -75,36 +107,55 @@ class PackageTreeNode:
 			except IOError:
 				pass
 
-		# depends.txt
-		import re
-		pattern = re.compile("^([a-z0-9_]+)\??$")
-		if not "depends" in result and not "optional_depends" in result:
-			try:
-				with open(self.baseDir + "/depends.txt", "r") as myfile:
-					contents = myfile.read()
-					soft = []
-					hard = []
-					for line in contents.split("\n"):
-						line = line.strip()
-						if pattern.match(line):
-							if line[len(line) - 1] == "?":
-								soft.append( line[:-1])
-							else:
-								hard.append(line)
+		# Read dependencies
+		if "depends" in result or "optional_depends" in result:
+			result["depends"] = get_csv_line(result.get("depends"))
+			result["optional_depends"] = get_csv_line(result.get("optional_depends"))
 
-					result["depends"] = hard
-					result["optional_depends"] = soft
+		elif os.path.isfile(self.baseDir + "/depends.txt"):
+			pattern = re.compile("^([a-z0-9_]+)\??$")
 
-			except IOError:
-				pass
+			with open(self.baseDir + "/depends.txt", "r") as myfile:
+				contents = myfile.read()
+				soft = []
+				hard = []
+				for line in contents.split("\n"):
+					line = line.strip()
+					if pattern.match(line):
+						if line[len(line) - 1] == "?":
+							soft.append( line[:-1])
+						else:
+							hard.append(line)
+
+				result["depends"] = hard
+				result["optional_depends"] = soft
 
 		else:
-			if "depends" in result:
-				result["depends"] = [x.strip() for x in result["depends"].split(",")]
-			if "optional_depends" in result:
-				result["optional_depends"] = [x.strip() for x in result["optional_depends"].split(",")]
+			result["depends"] = []
+			result["optional_depends"] = []
 
 
+		def checkDependencies(deps):
+			for dep in result["depends"]:
+				if not basenamePattern.match(dep):
+					if " " in dep:
+						raise MinetestCheckError("Invalid dependency name '{}' for mod at {}, did you forget a comma?" \
+							.format(dep, self.relative))
+					else:
+						raise MinetestCheckError(
+								"Invalid dependency name '{}' for mod at {}, names must only contain a-z0-9_." \
+									.format(dep, self.relative))
+
+
+		# Check dependencies
+		checkDependencies(result["depends"])
+		checkDependencies(result["optional_depends"])
+
+		# Fix games using "name" as "title"
+		if self.type == ContentType.GAME:
+			result["title"] = result["name"]
+			del result["name"]
+
 		# Calculate Title
 		if "name" in result and not "title" in result:
 			result["title"] = result["name"].replace("_", " ").title()
@@ -122,37 +173,58 @@ class PackageTreeNode:
 
 		self.meta = result
 
-	def add_children_from_mod_dir(self, dir):
+	def add_children_from_mod_dir(self, subdir):
+		dir = self.baseDir
+		relative = self.relative
+		if subdir:
+			dir += "/" + subdir
+			relative += subdir + "/"
+
 		for entry in next(os.walk(dir))[1]:
 			path = os.path.join(dir, entry)
 			if not entry.startswith('.') and os.path.isdir(path):
-				child = PackageTreeNode(path, self.relative + entry + "/", name=entry)
+				child = PackageTreeNode(path, relative + entry + "/", name=entry)
 				if not child.type.isModLike():
-					raise MinetestCheckError(("Expecting mod or modpack, found {} at {} inside {}") \
-							.format(child.type.value, child.relative, self.type.value))
+					raise MinetestCheckError("Expecting mod or modpack, found {} at {} inside {}" \
+						.format(child.type.value, child.relative, self.type.value))
+
+				if child.name is None:
+					raise MinetestCheckError("Missing base name for mod at {}".format(self.relative))
 
 				self.children.append(child)
 
+	def getModNames(self):
+		return self.fold("name", type=ContentType.MOD)
 
-	def fold(self, attr, key=None, acc=None):
-		if acc is None:
-			acc = set()
-
-		if self.meta is None:
-			return acc
-
-		at = getattr(self, attr)
-		value = at if key is None else at.get(key)
-
-		if isinstance(value, list):
-			acc |= set(value)
-		elif value is not None:
-			acc.add(value)
+	# attr: Attribute name
+	# key: Key in attribute
+	# retval: Accumulator
+	# type: Filter to type
+	def fold(self, attr, key=None, retval=None, type=None):
+		if retval is None:
+			retval = set()
 
+		# Iterate through children
 		for child in self.children:
-			child.fold(attr, key, acc)
+			child.fold(attr, key, retval, type)
 
-		return acc
+		# Filter on type
+		if type and type != self.type:
+			return retval
+
+		# Get attribute
+		at = getattr(self, attr)
+		if not at:
+			return retval
+
+		# Get value
+		value = at if key is None else at.get(key)
+		if isinstance(value, list):
+			retval |= set(value)
+		elif value:
+			retval.add(value)
+
+		return retval
 
 	def get(self, key):
 		return self.meta.get(key)

app/tasks/pkgtasks.py

@@ -1,17 +1,17 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
+# ContentDB
+# Copyright (C) 2018-21 rubenwardy
 #
 # This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
+# it under the terms of the GNU Affero General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# GNU Affero General Public License for more details.
 #
-# You should have received a copy of the GNU General Public License
+# You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
@@ -20,5 +20,10 @@ from app.tasks import celery
 
 @celery.task()
 def updatePackageScores():
-	Package.query.update({ "score": Package.score * 0.95 })
+	Package.query.update({ "score_downloads": Package.score_downloads * 0.95 })
+	db.session.commit()
+
+	for package in Package.query.all():
+		package.recalcScore()
+
 	db.session.commit()

app/template_filters.py

@@ -1,9 +1,33 @@
-from . import app
+from . import app, utils
+from .models import Permission, Package, PackageState, PackageRelease
+from .utils import abs_url_for, url_set_query
+from flask_login import current_user
+from flask_babel import format_timedelta, gettext
 from urllib.parse import urlparse
+from datetime import datetime as dt
+
+from .utils.markdown import get_headings
+
 
 @app.context_processor
 def inject_debug():
-    return dict(debug=app.debug)
+	return dict(debug=app.debug)
+
+@app.context_processor
+def inject_functions():
+	check_global_perm = Permission.checkPerm
+	return dict(abs_url_for=abs_url_for, url_set_query=url_set_query,
+			check_global_perm=check_global_perm,
+			get_headings=get_headings)
+
+@app.context_processor
+def inject_todo():
+	todo_list_count = None
+	if current_user and current_user.is_authenticated and current_user.canAccessTodoList():
+		todo_list_count = Package.query.filter_by(state=PackageState.READY_FOR_REVIEW).count()
+		todo_list_count += PackageRelease.query.filter_by(approved=False, task_id=None).count()
+
+	return dict(todo_list_count=todo_list_count)
 
 @app.template_filter()
 def throw(err):
@@ -15,8 +39,29 @@ def domain(url):
 
 @app.template_filter()
 def date(value):
-    return value.strftime("%Y-%m-%d")
+	return value.strftime("%Y-%m-%d")
+
+
+@app.template_filter()
+def full_datetime(value):
+	return value.strftime("%Y-%m-%d %H:%M") + " UTC"
 
 @app.template_filter()
 def datetime(value):
-    return value.strftime("%Y-%m-%d %H:%M") + " UTC"
+	delta = dt.utcnow() - value
+	if delta.days == 0:
+		return gettext("%(delta)s ago", delta=format_timedelta(value))
+	else:
+		return full_datetime(value)
+
+@app.template_filter()
+def isodate(value):
+	return value.strftime("%Y-%m-%d")
+
+@app.template_filter()
+def timedelta(value):
+	return format_timedelta(value)
+
+@app.template_filter()
+def abs_url(url):
+	return utils.abs_url(url)

app/templates/admin/audit.html

@@ -0,0 +1,16 @@
+{% extends "base.html" %}
+
+{% block title %}
+Audit Log
+{% endblock %}
+
+{% block content %}
+	<h1>Audit Log</h1>
+
+	{% from "macros/pagination.html" import render_pagination %}
+	{% from "macros/audit_log.html" import render_audit_log %}
+
+	{{ render_pagination(pagination, url_set_query) }}
+	{{ render_audit_log(log, current_user) }}
+	{{ render_pagination(pagination, url_set_query) }}
+{% endblock %}