Reduce chance of accidental release deletion

Fixes #176

.dockerignore

@@ -0,0 +1,5 @@
+.git
+data
+uploads
+*.pyc
+__pycache__

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,13 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: Unconfirmed Bug
+assignees: ''
+---
+
+## Summary
+Describe your problem here
+
+##### Steps to reproduce
+For bug reports or build issues, explain how the problem happened

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,25 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: Feature
+assignees: ''
+---
+
+## Problem
+
+A clear and concise description of what the problem is.
+ie: Why is this needed?
+Ex. I'm always frustrated when [...]
+
+## Solutions
+
+A clear and concise description of what you want to happen.
+
+## Alternatives
+
+A clear and concise description of any alternative solutions or features you've considered.
+
+## Additional context
+
+Add any other context or screenshots about the feature request here.

.github/ISSUE_TEMPLATE/policy.md

@@ -0,0 +1,7 @@
+---
+name: Policy suggestion
+about: Suggest a change to the guidelines
+title: ''
+labels: Policy
+assignees: ''
+---

.github/SECURITY.md

@@ -0,0 +1,19 @@
+# Security Policy
+
+## Supported Versions
+
+We only support the latest production version, deployed to <https://content.minetest.net>.
+See the [releases page](https://github.com/minetest/contentdb/releases).
+
+## Reporting a Vulnerability
+
+We ask that you report vulnerabilities privately, by contacting rubenwardy,
+to give us time to fix them. You can do that by using one of the methods outlined in the following link:
+
+* https://rubenwardy.com/contact/
+
+Depending on severity, we will either create a private issue for the vulnerability
+and release a security update, or give you permission to file the issue publicly.
+
+For more information on the justification of this policy, see
+[Responsible Disclosure](https://en.wikipedia.org/wiki/Responsible_disclosure).

.gitignore

@@ -1,12 +1,15 @@
-config.cfg
+/config.cfg
-config.prod.cfg
+/*.env
 *.sqlite
-main.css
+.vscode
+custom.css
 tmp
 log.txt
 *.rdb
-uploads
+app/public/uploads
-thumbnails
+app/public/thumbnails
+celerybeat-schedule
+/data
 
 # Created by https://www.gitignore.io/api/linux,macos,python,windows
 

.gitlab-ci.yml

@@ -0,0 +1,22 @@
+image: docker/compose
+services:
+  - docker:dind
+cache:
+  key: "$CI_COMMIT_REF_SLUG"
+  paths:
+    - /var/lib/docker
+
+# build:
+#   stage: build
+#   script:
+#     - cp utils/gitlabci/* .
+#     - docker-compose build
+
+UI_Test:
+  stage: test
+  script:
+    - cp utils/gitlabci/* .
+    - docker-compose up -d
+    - ./utils/run_migrations.sh
+    - ./utils/tests_cov.sh
+    - docker-compose down

Dockerfile

@@ -0,0 +1,22 @@
+FROM python:3.6
+
+RUN groupadd -g 5123 cdb && \
+    useradd -r -u 5123 -g cdb cdb
+
+WORKDIR /home/cdb
+
+RUN mkdir /var/cdb
+RUN chown -R cdb:cdb /var/cdb
+
+COPY requirements.txt requirements.txt
+RUN pip install -r requirements.txt
+RUN pip install gunicorn
+
+COPY utils utils
+COPY config.cfg config.cfg
+COPY migrations migrations
+COPY app app
+
+RUN chown -R cdb:cdb /home/cdb
+
+USER cdb

README.md

@@ -1,64 +1,63 @@
 # Content Database
 
-## Setup
+Content database for Minetest mods, games, and more.
 
-First create a Python virtual env:
+Developed by rubenwardy, license GPLv3.0+.
-
-	virtualenv env -ppython3
-	source env/bin/activate
-
-then use pip:
-
-	pip3 install -r requirements.txt
-
-### Development
-
-* Copy config.example.cfg to config.cfg
-* Fill SECRET_KEY and WTF_CSRF_SECRET_KEY in with a random string
-* Make a Github OAuth Client at <https://github.com/settings/developers>:
-	* Homepage URL - `http://localhost:5000/`
-	* Authorization callback URL - `http://localhost:5000/user/github/callback/`
-* Put client id and client secret in GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET
-* Setup the database: python3 setup.py
-
-
-## Running
-
-### Development
-
-You need to enter the virtual environment if you haven't yet in
-the current session:
-
-	source env/bin/activate
-
-If you need to, reset the db like so:
-
-	python3 setup.py -t
-
-Then run the server:
-
-	./rundebug.py
-
-Then view in your web browser: http://localhost:5000/
 
 ## How-tos
 
-### Start celery worker
+Note: you should first read one of the guides on the [Github repo wiki](https://github.com/minetest/contentdb/wiki)
 
 ```sh
+# Run celery worker
 FLASK_CONFIG=../config.cfg celery -A app.tasks.celery worker
-```
 
-### Create migration
-
-```sh
 # if sqlite
-python setup.py -t
+python utils/setup.py -t
 rm db.sqlite && python setup.py -t && FLASK_CONFIG=../config.cfg FLASK_APP=app/__init__.py flask db stamp head
 
 # Create migration
 FLASK_CONFIG=../config.cfg FLASK_APP=app/__init__.py flask db migrate
-
 # Run migration
 FLASK_CONFIG=../config.cfg FLASK_APP=app/__init__.py flask db upgrade
+
+# Enter docker
+docker exec -it contentdb_app_1 bash
+
+# Hot/live reload (only works with FLASK_DEBUG=1)
+./utils/reload.sh
+
+# Cold update a running version of CDB with minimal downtime
+./utils/update.sh
+```
+
+## Database
+
+
+```mermaid
+classDiagram
+
+User "1" --> "*" Package
+User --> UserEmailVerification
+User "1" --> "*" Notification
+Package "1" --> "*" Release
+Package "1" --> "*" Dependency
+Package "1" --> "*" Tag
+Package "1" --> "*" MetaPackage : provides
+Release --> MinetestVersion
+Package --> License
+Dependency --> Package
+Dependency --> MetaPackage
+MetaPackage "1" --> "*" Package
+Package "1" --> "*" Screenshot
+Package "1" --> "*" Thread
+Thread "1" --> "*" Reply
+Thread "1" --> "*" User : watchers
+User "1" --> "*" Thread
+User "1" --> "*" Reply
+User "1" --> "*" ForumTopic
+
+User --> "0..1" EmailPreferences
+User "1" --> "*" APIToken
+APIToken --> Package
 ```

app/__init__.py

@@ -17,25 +17,77 @@
 
 from flask import *
 from flask_user import *
+from flask_gravatar import Gravatar
 import flask_menu as menu
 from flask_mail import Mail
-from flask.ext import markdown
+from flaskext.markdown import Markdown
 from flask_github import GitHub
 from flask_wtf.csrf import CsrfProtect
 from flask_flatpages import FlatPages
-import os
+from flask_babel import Babel
+import os, redis
 
 app = Flask(__name__, static_folder="public/static")
 app.config["FLATPAGES_ROOT"] = "flatpages"
 app.config["FLATPAGES_EXTENSION"] = ".md"
 app.config.from_pyfile(os.environ["FLASK_CONFIG"])
 
+r = redis.Redis.from_url(app.config["REDIS_URL"])
+
 menu.Menu(app=app)
-markdown.Markdown(app, extensions=["fenced_code"], safe_mode=True, output_format="html5")
+markdown = Markdown(app, extensions=["fenced_code"], safe_mode=True, output_format="html5")
 github = GitHub(app)
 csrf = CsrfProtect(app)
 mail = Mail(app)
 pages = FlatPages(app)
+babel = Babel(app)
+gravatar = Gravatar(app,
+		size=58,
+		rating='g',
+		default='mp',
+		force_default=False,
+		force_lower=False,
+		use_ssl=True,
+		base_url=None)
 
-from . import models, tasks
+from .sass import sass
-from .views import *
+sass(app)
+
+
+if not app.debug and app.config["MAIL_UTILS_ERROR_SEND_TO"]:
+	from .maillogger import register_mail_error_handler
+	register_mail_error_handler(app, mail)
+
+
+@babel.localeselector
+def get_locale():
+	return request.accept_languages.best_match(app.config['LANGUAGES'].keys())
+
+from . import models, tasks, template_filters
+
+from .blueprints import create_blueprints
+create_blueprints(app)
+
+from flask_login import logout_user
+
+@app.route("/uploads/<path:path>")
+def send_upload(path):
+	return send_from_directory(app.config['UPLOAD_DIR'], path)
+
+@menu.register_menu(app, ".help", "Help", order=19, endpoint_arguments_constructor=lambda: { 'path': 'help' })
+@app.route('/<path:path>/')
+def flatpage(path):
+    page = pages.get_or_404(path)
+    template = page.meta.get('template', 'flatpage.html')
+    return render_template(template, page=page)
+
+@app.before_request
+def check_for_ban():
+	if current_user.is_authenticated:
+		if current_user.rank == models.UserRank.BANNED:
+			flash("You have been banned.", "error")
+			logout_user()
+			return redirect(url_for('user.login'))
+		elif current_user.rank == models.UserRank.NOT_JOINED:
+			current_user.rank = models.UserRank.MEMBER
+			models.db.session.commit()

app/blueprints/__init__.py

@@ -0,0 +1,10 @@
+import os, importlib
+
+def create_blueprints(app):
+	dir = os.path.dirname(os.path.realpath(__file__))
+	modules = next(os.walk(dir))[1]	
+	
+	for modname in modules:		
+		if all(c.islower() for c in modname):
+			module = importlib.import_module("." + modname, __name__)		
+			app.register_blueprint(module.bp)

app/blueprints/admin/__init__.py

@@ -15,21 +15,8 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-from flask import *
+from flask import Blueprint
-from flask_user import *
-from app import app
-from app.models import *
-from app.utils import is_package_page
-from .packages import build_packages_query
 
-@app.route("/api/packages/")
+bp = Blueprint("admin", __name__)
-def api_packages_page():
-	query, _ = build_packages_query()
-	pkgs = [package.getAsDictionaryShort(app.config["BASE_URL"]) \
-			for package in query.all() if package.getDownloadRelease() is not None]
-	return jsonify(pkgs)
 
-@app.route("/api/packages/<author>/<name>/")
+from . import admin, licenseseditor, tagseditor, versioneditor
-@is_package_page
-def api_package_page(package):
-	return jsonify(package.getAsDictionary(app.config["BASE_URL"]))

app/blueprints/admin/admin.py

@@ -17,26 +17,47 @@
 
 from flask import *
 from flask_user import *
-from flask.ext import menu
+import flask_menu as menu
-from app import app
+from . import bp
 from app.models import *
-from app.tasks.importtasks import importRepoScreenshot, importAllDependencies
+from celery import uuid, group
-from app.tasks.forumtasks  import importUsersFromModList, importKrocksModList
+from app.tasks.importtasks import importRepoScreenshot, importAllDependencies, makeVCSRelease, checkZipRelease
+from app.tasks.forumtasks  import importTopicList, checkAllForumAccounts
 from flask_wtf import FlaskForm
 from wtforms import *
-from app.utils import loginUser, rank_required
+from app.utils import loginUser, rank_required, triggerNotif
+import datetime
 
-@app.route("/admin/", methods=["GET", "POST"])
+@bp.route("/admin/", methods=["GET", "POST"])
 @rank_required(UserRank.ADMIN)
 def admin_page():
 	if request.method == "POST":
 		action = request.form["action"]
-		if action == "importusers":
+		if action == "delstuckreleases":
-			task = importUsersFromModList.delay()
+			PackageRelease.query.filter(PackageRelease.task_id != None).delete()
-			return redirect(url_for("check_task", id=task.id, r=url_for("user_list_page")))
+			db.session.commit()
+			return redirect(url_for("admin.admin_page"))
+		elif action == "checkreleases":
+			releases = PackageRelease.query.filter(PackageRelease.url.like("/uploads/%")).all()
+
+			tasks = []
+			for release in releases:
+				zippath = release.url.replace("/uploads/", app.config["UPLOAD_DIR"])
+				tasks.append(checkZipRelease.s(release.id, zippath))
+
+			result = group(tasks).apply_async()
+
+			while not result.ready():
+				import time
+				time.sleep(0.1)
+
+			return redirect(url_for("todo.view"))
 		elif action == "importmodlist":
-			task = importKrocksModList.delay()
+			task = importTopicList.delay()
-			return redirect(url_for("check_task", id=task.id, r=url_for("todo_topics_page")))
+			return redirect(url_for("tasks.check", id=task.id, r=url_for("todo.topics")))
+		elif action == "checkusers":
+			task = checkAllForumAccounts.delay()
+			return redirect(url_for("tasks.check", id=task.id, r=url_for("admin.admin_page")))
 		elif action == "importscreenshots":
 			packages = Package.query \
 				.filter_by(soft_deleted=False) \
@@ -46,7 +67,7 @@ def admin_page():
 			for package in packages:
 				importRepoScreenshot.delay(package.id)
 
-			return redirect(url_for("admin_page"))
+			return redirect(url_for("admin.admin_page"))
 		elif action == "restore":
 			package = Package.query.get(request.form["package"])
 			if package is None:
@@ -54,10 +75,10 @@ def admin_page():
 			else:
 				package.soft_deleted = False
 				db.session.commit()
-				return redirect(url_for("admin_page"))
+				return redirect(url_for("admin.admin_page"))
 		elif action == "importdepends":
 			task = importAllDependencies.delay()
-			return redirect(url_for("check_task", id=task.id, r=url_for("admin_page")))
+			return redirect(url_for("tasks.check", id=task.id, r=url_for("admin.admin_page")))
 		elif action == "modprovides":
 			packages = Package.query.filter_by(type=PackageType.MOD).all()
 			mpackage_cache = {}
@@ -66,7 +87,32 @@ def admin_page():
 					p.provides.append(MetaPackage.GetOrCreate(p.name, mpackage_cache))
 
 			db.session.commit()
-			return redirect(url_for("admin_page"))
+			return redirect(url_for("admin.admin_page"))
+		elif action == "recalcscores":
+			for p in Package.query.all():
+				p.setStartScore()
+
+			db.session.commit()
+			return redirect(url_for("admin.admin_page"))
+		elif action == "vcsrelease":
+			for package in Package.query.filter(Package.repo.isnot(None)).all():
+				if package.releases.count() != 0:
+					continue
+
+				rel = PackageRelease()
+				rel.package  = package
+				rel.title    = datetime.date.today().isoformat()
+				rel.url      = ""
+				rel.task_id  = uuid()
+				rel.approved = True
+				db.session.add(rel)
+				db.session.commit()
+
+				makeVCSRelease.apply_async((rel.id, "master"), task_id=rel.task_id)
+
+				msg = "{}: Release {} created".format(package.title, rel.title)
+				triggerNotif(package.author, current_user, msg, rel.getEditURL())
+				db.session.commit()
 
 		else:
 			flash("Unknown action: " + action, "error")
@@ -79,19 +125,19 @@ class SwitchUserForm(FlaskForm):
 	submit = SubmitField("Switch")
 
 
-@app.route("/admin/switchuser/", methods=["GET", "POST"])
+@bp.route("/admin/switchuser/", methods=["GET", "POST"])
 @rank_required(UserRank.ADMIN)
-def switch_user_page():
+def switch_user():
 	form = SwitchUserForm(formdata=request.form)
 	if request.method == "POST" and form.validate():
 		user = User.query.filter_by(username=form["username"].data).first()
 		if user is None:
 			flash("Unable to find user", "error")
 		elif loginUser(user):
-			return redirect(url_for("user_profile_page", username=current_user.username))
+			return redirect(url_for("users.profile", username=current_user.username))
 		else:
 			flash("Unable to login as user", "error")
 
 
 	# Process GET or invalid POST
-	return render_template("admin/switch_user_page.html", form=form)
+	return render_template("admin/switch_user.html", form=form)

app/blueprints/admin/licenseseditor.py

@@ -0,0 +1,62 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_user import *
+from . import bp
+from app.models import *
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+from app.utils import rank_required
+
+@bp.route("/licenses/")
+@rank_required(UserRank.MODERATOR)
+def license_list():
+	return render_template("admin/licenses/list.html", licenses=License.query.order_by(db.asc(License.name)).all())
+
+class LicenseForm(FlaskForm):
+	name	 = StringField("Name", [InputRequired(), Length(3,100)])
+	is_foss  = BooleanField("Is FOSS")
+	submit   = SubmitField("Save")
+
+@bp.route("/licenses/new/", methods=["GET", "POST"])
+@bp.route("/licenses/<name>/edit/", methods=["GET", "POST"])
+@rank_required(UserRank.MODERATOR)
+def create_edit_license(name=None):
+	license = None
+	if name is not None:
+		license = License.query.filter_by(name=name).first()
+		if license is None:
+			abort(404)
+
+	form = LicenseForm(formdata=request.form, obj=license)
+	if request.method == "GET" and license is None:
+		form.is_foss.data = True
+	elif request.method == "POST" and form.validate():
+		if license is None:
+			license = License(form.name.data)
+			db.session.add(license)
+			flash("Created license " + form.name.data, "success")
+		else:
+			flash("Updated license " + form.name.data, "success")
+
+		form.populate_obj(license)
+		db.session.commit()
+		return redirect(url_for("admin.license_list"))
+
+	return render_template("admin/licenses/edit.html", license=license, form=form)

app/blueprints/admin/tagseditor.py

@@ -17,27 +17,27 @@
 
 from flask import *
 from flask_user import *
-from app import app
+from . import bp
 from app.models import *
 from flask_wtf import FlaskForm
 from wtforms import *
 from wtforms.validators import *
 from app.utils import rank_required
 
-@app.route("/tags/")
+@bp.route("/tags/")
 @rank_required(UserRank.MODERATOR)
-def tag_list_page():
+def tag_list():
-	return render_template("tags/list.html", tags=Tag.query.order_by(db.asc(Tag.title)).all())
+	return render_template("admin/tags/list.html", tags=Tag.query.order_by(db.asc(Tag.title)).all())
 
 class TagForm(FlaskForm):
 	title	 = StringField("Title", [InputRequired(), Length(3,100)])
 	name     = StringField("Name", [Optional(), Length(1, 20), Regexp("^[a-z0-9_]", 0, "Lower case letters (a-z), digits (0-9), and underscores (_) only")])
 	submit   = SubmitField("Save")
 
-@app.route("/tags/new/", methods=["GET", "POST"])
+@bp.route("/tags/new/", methods=["GET", "POST"])
-@app.route("/tags/<name>/edit/", methods=["GET", "POST"])
+@bp.route("/tags/<name>/edit/", methods=["GET", "POST"])
 @rank_required(UserRank.MODERATOR)
-def createedit_tag_page(name=None):
+def create_edit_tag(name=None):
 	tag = None
 	if name is not None:
 		tag = Tag.query.filter_by(name=name).first()
@@ -52,6 +52,6 @@ def createedit_tag_page(name=None):
 		else:
 			form.populate_obj(tag)
 		db.session.commit()
-		return redirect(url_for("createedit_tag_page", name=tag.name))
+		return redirect(url_for("admin.create_edit_tag", name=tag.name))
 
-	return render_template("tags/edit.html", tag=tag, form=form)
+	return render_template("admin/tags/edit.html", tag=tag, form=form)

app/blueprints/admin/versioneditor.py

@@ -0,0 +1,60 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_user import *
+from . import bp
+from app.models import *
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+from app.utils import rank_required
+
+@bp.route("/versions/")
+@rank_required(UserRank.MODERATOR)
+def version_list():
+	return render_template("admin/versions/list.html", versions=MinetestRelease.query.order_by(db.asc(MinetestRelease.id)).all())
+
+class VersionForm(FlaskForm):
+	name	 = StringField("Name", [InputRequired(), Length(3,100)])
+	protocol = IntegerField("Protocol")
+	submit   = SubmitField("Save")
+
+@bp.route("/versions/new/", methods=["GET", "POST"])
+@bp.route("/versions/<name>/edit/", methods=["GET", "POST"])
+@rank_required(UserRank.MODERATOR)
+def create_edit_version(name=None):
+	version = None
+	if name is not None:
+		version = MinetestRelease.query.filter_by(name=name).first()
+		if version is None:
+			abort(404)
+
+	form = VersionForm(formdata=request.form, obj=version)
+	if request.method == "POST" and form.validate():
+		if version is None:
+			version = MinetestRelease(form.name.data)
+			db.session.add(version)
+			flash("Created version " + form.name.data, "success")
+		else:
+			flash("Updated version " + form.name.data, "success")
+
+		form.populate_obj(version)
+		db.session.commit()
+		return redirect(url_for("admin.version_list"))
+
+	return render_template("admin/versions/edit.html", version=version, form=form)

app/blueprints/api/__init__.py

@@ -0,0 +1,21 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+from flask import Blueprint
+
+bp = Blueprint("api", __name__)
+
+from . import tokens, endpoints

app/blueprints/api/auth.py

@@ -0,0 +1,42 @@
+# Content DB
+# Copyright (C) 2019  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+from flask import request, make_response, jsonify, abort
+from app.models import APIToken
+from functools import wraps
+
+def is_api_authd(f):
+	@wraps(f)
+	def decorated_function(*args, **kwargs):
+		token = None
+
+		value = request.headers.get("authorization")
+		if value is None:
+			pass
+		elif value[0:7].lower() == "bearer ":
+			access_token = value[7:]
+			if len(access_token) < 10:
+				abort(400)
+
+			token = APIToken.query.filter_by(access_token=access_token).first()
+			if token is None:
+				abort(403)
+		else:
+			abort(403)
+
+		return f(token=token, *args, **kwargs)
+
+	return decorated_function

app/blueprints/api/endpoints.py

@@ -0,0 +1,109 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_user import *
+from . import bp
+from .auth import is_api_authd
+from app.models import *
+from app.utils import is_package_page
+from app.querybuilder import QueryBuilder
+
+@bp.route("/api/packages/")
+def packages():
+	qb    = QueryBuilder(request.args)
+	query = qb.buildPackageQuery()
+	ver   = qb.getMinetestVersion()
+
+	pkgs = [package.getAsDictionaryShort(current_app.config["BASE_URL"], version=ver) \
+			for package in query.all()]
+	return jsonify(pkgs)
+
+
+@bp.route("/api/packages/<author>/<name>/")
+@is_package_page
+def package(package):
+	return jsonify(package.getAsDictionary(current_app.config["BASE_URL"]))
+
+
+@bp.route("/api/packages/<author>/<name>/dependencies/")
+@is_package_page
+def package_dependencies(package):
+	ret = []
+
+	for dep in package.dependencies:
+		name = None
+		fulfilled_by = None
+
+		if dep.package:
+			name = dep.package.name
+			fulfilled_by = [ dep.package.getAsDictionaryKey() ]
+
+		elif dep.meta_package:
+			name = dep.meta_package.name
+			fulfilled_by = [ pkg.getAsDictionaryKey() for pkg in dep.meta_package.packages]
+
+		else:
+			raise "Malformed dependency"
+
+		ret.append({
+			"name": name,
+			"is_optional": dep.optional,
+			"packages": fulfilled_by
+		})
+
+	return jsonify(ret)
+
+
+@bp.route("/api/topics/")
+def topics():
+	qb     = QueryBuilder(request.args)
+	query  = qb.buildTopicQuery(show_added=True)
+	return jsonify([t.getAsDictionary() for t in query.all()])
+
+
+@bp.route("/api/topic_discard/", methods=["POST"])
+@login_required
+def topic_set_discard():
+	tid = request.args.get("tid")
+	discard = request.args.get("discard")
+	if tid is None or discard is None:
+		abort(400)
+
+	topic = ForumTopic.query.get(tid)
+	if not topic.checkPerm(current_user, Permission.TOPIC_DISCARD):
+		abort(403)
+
+	topic.discarded = discard == "true"
+	db.session.commit()
+
+	return jsonify(topic.getAsDictionary())
+
+
+@bp.route("/api/minetest_versions/")
+def versions():
+	return jsonify([{ "name": rel.name, "protocol_version": rel.protocol }\
+			for rel in MinetestRelease.query.all() if rel.getActual() is not None])
+
+
+@bp.route("/api/whoami/")
+@is_api_authd
+def whoami(token):
+	if token is None:
+		return jsonify({ "is_authenticated": False, "username": None })
+	else:
+		return jsonify({ "is_authenticated": True, "username": token.owner.username })

app/blueprints/api/tokens.py

@@ -0,0 +1,141 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import render_template, redirect, request, session, url_for, abort
+from flask_user import login_required, current_user
+from . import bp
+from app.models import db, User, APIToken, Package, Permission
+from app.utils import randomString
+from app.querybuilder import QueryBuilder
+
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+from wtforms.ext.sqlalchemy.fields import QuerySelectField
+
+class CreateAPIToken(FlaskForm):
+	name	     = StringField("Name", [InputRequired(), Length(1, 30)])
+	submit	     = SubmitField("Save")
+
+
+@bp.route("/users/<username>/tokens/")
+@login_required
+def list_tokens(username):
+	user = User.query.filter_by(username=username).first()
+	if user is None:
+		abort(404)
+
+	if not user.checkPerm(current_user, Permission.CREATE_TOKEN):
+		abort(403)
+
+	return render_template("api/list_tokens.html", user=user)
+
+
+@bp.route("/users/<username>/tokens/new/", methods=["GET", "POST"])
+@bp.route("/users/<username>/tokens/<int:id>/edit/", methods=["GET", "POST"])
+@login_required
+def create_edit_token(username, id=None):
+	user = User.query.filter_by(username=username).first()
+	if user is None:
+		abort(404)
+
+	if not user.checkPerm(current_user, Permission.CREATE_TOKEN):
+		abort(403)
+
+	is_new = id is None
+
+	token = None
+	access_token = None
+	if not is_new:
+		token = APIToken.query.get(id)
+		if token is None:
+			abort(404)
+		elif token.owner != user:
+			abort(403)
+
+		access_token = session.pop("token_" + str(id), None)
+
+	form = CreateAPIToken(formdata=request.form, obj=token)
+	if request.method == "POST" and form.validate():
+		if is_new:
+			token = APIToken()
+			token.owner = user
+			token.access_token = randomString(32)
+
+		form.populate_obj(token)
+		db.session.add(token)
+
+		db.session.commit() # save
+
+		# Store token so it can be shown in the edit page
+		session["token_" + str(token.id)] = token.access_token
+
+		return redirect(url_for("api.create_edit_token", username=username, id=token.id))
+
+	return render_template("api/create_edit_token.html", user=user, form=form, token=token, access_token=access_token)
+
+
+@bp.route("/users/<username>/tokens/<int:id>/reset/", methods=["POST"])
+@login_required
+def reset_token(username, id):
+	user = User.query.filter_by(username=username).first()
+	if user is None:
+		abort(404)
+
+	if not user.checkPerm(current_user, Permission.CREATE_TOKEN):
+		abort(403)
+
+	is_new = id is None
+
+	token = APIToken.query.get(id)
+	if token is None:
+		abort(404)
+	elif token.owner != user:
+		abort(403)
+
+	token.access_token = randomString(32)
+
+	db.session.commit() # save
+
+	# Store token so it can be shown in the edit page
+	session["token_" + str(token.id)] = token.access_token
+
+	return redirect(url_for("api.create_edit_token", username=username, id=token.id))
+
+
+@bp.route("/users/<username>/tokens/<int:id>/delete/", methods=["POST"])
+@login_required
+def delete_token(username, id):
+	user = User.query.filter_by(username=username).first()
+	if user is None:
+		abort(404)
+
+	if not user.checkPerm(current_user, Permission.CREATE_TOKEN):
+		abort(403)
+
+	is_new = id is None
+
+	token = APIToken.query.get(id)
+	if token is None:
+		abort(404)
+	elif token.owner != user:
+		abort(403)
+
+	db.session.delete(token)
+	db.session.commit()
+
+	return redirect(url_for("api.list_tokens", username=username))

app/blueprints/homepage/__init__.py

@@ -0,0 +1,21 @@
+from flask import Blueprint, render_template
+
+bp = Blueprint("homepage", __name__)
+
+from app.models import *
+import flask_menu as menu
+from sqlalchemy.sql.expression import func
+
+@bp.route("/")
+@menu.register_menu(bp, ".", "Home")
+def home():
+	query   = Package.query.filter_by(approved=True, soft_deleted=False)
+	count   = query.count()
+	new     = query.order_by(db.desc(Package.created_at)).limit(8).all()
+	pop_mod = query.filter_by(type=PackageType.MOD).order_by(db.desc(Package.score)).limit(8).all()
+	pop_gam = query.filter_by(type=PackageType.GAME).order_by(db.desc(Package.score)).limit(4).all()
+	pop_txp = query.filter_by(type=PackageType.TXP).order_by(db.desc(Package.score)).limit(4).all()
+	downloads_result = db.session.query(func.sum(PackageRelease.downloads)).one_or_none()
+	downloads = 0 if not downloads_result or not downloads_result[0] else downloads_result[0]
+	return render_template("index.html", count=count, downloads=downloads, \
+			new=new, pop_mod=pop_mod, pop_txp=pop_txp, pop_gam=pop_gam)

app/blueprints/metapackages/__init__.py

@@ -16,17 +16,19 @@
 
 
 from flask import *
+
+bp = Blueprint("metapackages", __name__)
+
 from flask_user import *
-from app import app
 from app.models import *
 
-@app.route("/metapackages/")
+@bp.route("/metapackages/")
-def meta_package_list_page():
+def list_all():
 	mpackages = MetaPackage.query.order_by(db.asc(MetaPackage.name)).all()
 	return render_template("meta/list.html", mpackages=mpackages)
 
-@app.route("/metapackages/<name>/")
+@bp.route("/metapackages/<name>/")
-def meta_package_page(name):
+def view(name):
 	mpackage = MetaPackage.query.filter_by(name=name).first()
 	if mpackage is None:
 		abort(404)

app/blueprints/notifications/__init__.py

@@ -15,19 +15,20 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-from flask import *
+from flask import Blueprint, render_template, redirect, url_for
 from flask_user import current_user, login_required
-from app import app
+from app.models import db
-from app.models import *
 
-@app.route("/notifications/")
+bp = Blueprint("notifications", __name__)
+
+@bp.route("/notifications/")
 @login_required
-def notifications_page():
+def list_all():
 	return render_template("notifications/list.html")
 
-@app.route("/notifications/clear/", methods=["POST"])
+@bp.route("/notifications/clear/", methods=["POST"])
 @login_required
-def clear_notifications_page():
+def clear():
 	current_user.notifications.clear()
 	db.session.commit()
-	return redirect(url_for("notifications_page"))
+	return redirect(url_for("notifications.list_all"))

app/blueprints/packages/__init__.py

@@ -0,0 +1,21 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+from flask import Blueprint
+
+bp = Blueprint("packages", __name__)
+
+from . import packages, screenshots, releases

app/blueprints/packages/editrequests.py

@@ -14,10 +14,8 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
-
 from flask import *
 from flask_user import *
-from flask.ext import menu
 from app import app
 from app.models import *
 

app/blueprints/packages/packages.py

@@ -15,114 +15,144 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-from flask import *
+from flask import render_template, abort, request, redirect, url_for, flash
-from flask_user import *
+from flask_user import current_user
-from flask.ext import menu
+import flask_menu as menu
-from app import app
-from app.models import *
-from app.tasks.importtasks import importRepoScreenshot, makeVCSRelease
 
+from . import bp
+
+from app.models import *
+from app.querybuilder import QueryBuilder
+from app.tasks.importtasks import importRepoScreenshot
 from app.utils import *
 
-from celery import uuid
 from flask_wtf import FlaskForm
 from wtforms import *
 from wtforms.validators import *
 from wtforms.ext.sqlalchemy.fields import QuerySelectField, QuerySelectMultipleField
+from sqlalchemy import or_
 
-def build_packages_query():
-	type_name = request.args.get("type")
-	type = None
-	if type_name is not None:
-		type = PackageType[type_name.upper()]
 
-	title = "Packages"
+@menu.register_menu(bp, ".mods", "Mods", order=11, endpoint_arguments_constructor=lambda: { 'type': 'mod' })
-	query = Package.query.filter_by(soft_deleted=False)
+@menu.register_menu(bp, ".games", "Games", order=12, endpoint_arguments_constructor=lambda: { 'type': 'game' })
+@menu.register_menu(bp, ".txp", "Texture Packs", order=13, endpoint_arguments_constructor=lambda: { 'type': 'txp' })
+@menu.register_menu(bp, ".random", "Random", order=14, endpoint_arguments_constructor=lambda: { 'random': '1', 'lucky': '1' })
+@bp.route("/packages/")
+def list_all():
+	qb    = QueryBuilder(request.args)
+	query = qb.buildPackageQuery()
+	title = qb.title
 
-	if type is not None:
+	if qb.lucky:
-		title = type.value + "s"
+		package = query.first()
-		query = query.filter_by(type=type, approved=True)
+		if package:
+			return redirect(package.getDetailsURL())
 
-	search = request.args.get("q")
+		topic = qb.buildTopicQuery().first()
-	if search is not None and search.strip() != "":
+		if qb.search and topic:
-		query = query.filter(Package.title.ilike('%' + search + '%'))
+			return redirect("https://forum.minetest.net/viewtopic.php?t=" + str(topic.topic_id))
 
-	return query, title
+	page  = get_int_or_abort(request.args.get("page"), 1)
-
+	num   = min(40, get_int_or_abort(request.args.get("n"), 100))
-@menu.register_menu(app, ".mods", "Mods", order=11, endpoint_arguments_constructor=lambda: { 'type': 'mod' })
-@menu.register_menu(app, ".games", "Games", order=12, endpoint_arguments_constructor=lambda: { 'type': 'game' })
-@menu.register_menu(app, ".txp", "Texture Packs", order=13, endpoint_arguments_constructor=lambda: { 'type': 'txp' })
-@app.route("/packages/")
-def packages_page():
-	if shouldReturnJson():
-		return redirect(url_for("api_packages_page"))
-
-	query, title = build_packages_query()
-	page  = int(request.args.get("page") or 1)
-	num   = min(42, int(request.args.get("n") or 100))
 	query = query.paginate(page, num, True)
 
 	search = request.args.get("q")
 	type_name = request.args.get("type")
 
-	next_url = url_for("packages_page", type=type_name, q=search, page=query.next_num) \
+	next_url = url_for("packages.list_all", type=type_name, q=search, page=query.next_num) \
 			if query.has_next else None
-	prev_url = url_for("packages_page", type=type_name, q=search, page=query.prev_num) \
+	prev_url = url_for("packages.list_all", type=type_name, q=search, page=query.prev_num) \
 			if query.has_prev else None
 
+	topics = None
+	if qb.search and not query.has_next:
+		topics = qb.buildTopicQuery().all()
+
 	tags = Tag.query.all()
-	return render_template("packages/list.html", title=title, packages=query.items, \
+	return render_template("packages/list.html", \
+			title=title, packages=query.items, topics=topics, \
 			query=search, tags=tags, type=type_name, \
 			next_url=next_url, prev_url=prev_url, page=page, page_max=query.pages, packages_count=query.total)
 
 
 def getReleases(package):
 	if package.checkPerm(current_user, Permission.MAKE_RELEASE):
-		return package.releases
+		return package.releases.limit(5)
 	else:
-		return [rel for rel in package.releases if rel.approved]
+		return package.releases.filter_by(approved=True).limit(5)
 
 
-@app.route("/packages/<author>/<name>/")
+@bp.route("/packages/<author>/<name>/")
 @is_package_page
-def package_page(package):
+def view(package):
 	clearNotifications(package.getDetailsURL())
 
 	alternatives = None
 	if package.type == PackageType.MOD:
 		alternatives = Package.query \
-				.filter_by(name=package.name, type=PackageType.MOD, soft_deleted=False) \
+			.filter_by(name=package.name, type=PackageType.MOD, soft_deleted=False) \
-				.filter(Package.id != package.id) \
+			.filter(Package.id != package.id) \
-				.order_by(db.asc(Package.title)) \
+			.order_by(db.desc(Package.score)) \
-				.all()
+			.all()
+
 
 	show_similar_topics = current_user == package.author or \
 			package.checkPerm(current_user, Permission.APPROVE_NEW)
 
 	similar_topics = None if not show_similar_topics else \
-			KrockForumTopic.query \
+			ForumTopic.query \
 				.filter_by(name=package.name) \
-				.filter(KrockForumTopic.topic_id != package.forums) \
+				.filter(ForumTopic.topic_id != package.forums) \
-				.filter(~ db.exists().where(Package.forums==KrockForumTopic.topic_id)) \
+				.filter(~ db.exists().where(Package.forums==ForumTopic.topic_id)) \
-				.order_by(db.asc(KrockForumTopic.name), db.asc(KrockForumTopic.title)) \
+				.order_by(db.asc(ForumTopic.name), db.asc(ForumTopic.title)) \
 				.all()
 
 	releases = getReleases(package)
 	requests = [r for r in package.requests if r.status == 0]
 
-	review_thread = Thread.query.filter_by(package_id=package.id, private=True).first()
+	review_thread = package.review_thread
 	if review_thread is not None and not review_thread.checkPerm(current_user, Permission.SEE_THREAD):
 		review_thread = None
 
+	topic_error = None
+	topic_error_lvl = "warning"
+	if not package.approved and package.forums is not None:
+		errors = []
+		if Package.query.filter_by(forums=package.forums, soft_deleted=False).count() > 1:
+			errors.append("<b>Error: Another package already uses this forum topic!</b>")
+			topic_error_lvl = "danger"
+
+		topic = ForumTopic.query.get(package.forums)
+		if topic is not None:
+			if topic.author != package.author:
+				errors.append("<b>Error: Forum topic author doesn't match package author.</b>")
+				topic_error_lvl = "danger"
+
+			if topic.wip:
+				errors.append("Warning: Forum topic is in WIP section, make sure package meets playability standards.")
+		elif package.type != PackageType.TXP:
+			errors.append("Warning: Forum topic not found. This may happen if the topic has only just been created.")
+
+		topic_error = "<br />".join(errors)
+
+
+	threads = Thread.query.filter_by(package_id=package.id)
+	if not current_user.is_authenticated:
+		threads = threads.filter_by(private=False)
+	elif not current_user.rank.atLeast(UserRank.EDITOR) and not current_user == package.author:
+		threads = threads.filter(or_(Thread.private == False, Thread.author == current_user))
+
+
 	return render_template("packages/view.html", \
 			package=package, releases=releases, requests=requests, \
 			alternatives=alternatives, similar_topics=similar_topics, \
-			review_thread=review_thread)
+			review_thread=review_thread, topic_error=topic_error, topic_error_lvl=topic_error_lvl, \
+			threads=threads.all())
 
 
-@app.route("/packages/<author>/<name>/download/")
+@bp.route("/packages/<author>/<name>/download/")
 @is_package_page
-def package_download_page(package):
+def download(package):
 	release = package.getDownloadRelease()
 
 	if release is None:
@@ -133,31 +163,36 @@ def package_download_page(package):
 			flash("No download available.", "error")
 			return redirect(package.getDetailsURL())
 	else:
+		PackageRelease.query.filter_by(id=release.id).update({
+				"downloads": PackageRelease.downloads + 1
+			})
+		db.session.commit()
+
 		return redirect(release.url, code=302)
 
 
 class PackageForm(FlaskForm):
 	name          = StringField("Name (Technical)", [InputRequired(), Length(1, 20), Regexp("^[a-z0-9_]", 0, "Lower case letters (a-z), digits (0-9), and underscores (_) only")])
 	title         = StringField("Title (Human-readable)", [InputRequired(), Length(3, 50)])
-	shortDesc     = StringField("Short Description (Plaintext)", [InputRequired(), Length(1,200)])
+	short_desc     = StringField("Short Description (Plaintext)", [InputRequired(), Length(1,200)])
 	desc          = TextAreaField("Long Description (Markdown)", [Optional(), Length(0,10000)])
 	type          = SelectField("Type", [InputRequired()], choices=PackageType.choices(), coerce=PackageType.coerce, default=PackageType.MOD)
-	license       = QuerySelectField("License", [InputRequired()], query_factory=lambda: License.query, get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	license       = QuerySelectField("License", [InputRequired()], query_factory=lambda: License.query.order_by(db.asc(License.name)), get_pk=lambda a: a.id, get_label=lambda a: a.name)
-	media_license = QuerySelectField("Media License", [InputRequired()], query_factory=lambda: License.query, get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	media_license = QuerySelectField("Media License", [InputRequired()], query_factory=lambda: License.query.order_by(db.asc(License.name)), get_pk=lambda a: a.id, get_label=lambda a: a.name)
-	provides_str  = StringField("Provides (mods included in package)", [Optional(), Length(0,1000)])
+	provides_str  = StringField("Provides (mods included in package)", [Optional()])
 	tags          = QuerySelectMultipleField('Tags', query_factory=lambda: Tag.query.order_by(db.asc(Tag.name)), get_pk=lambda a: a.id, get_label=lambda a: a.title)
-	harddep_str   = StringField("Hard Dependencies", [Optional(), Length(0,1000)])
+	harddep_str   = StringField("Hard Dependencies", [Optional()])
-	softdep_str   = StringField("Soft Dependencies", [Optional(), Length(0,1000)])
+	softdep_str   = StringField("Soft Dependencies", [Optional()])
-	repo          = StringField("VCS Repository URL", [Optional(), URL()])
+	repo          = StringField("VCS Repository URL", [Optional(), URL()], filters = [lambda x: x or None])
-	website       = StringField("Website URL", [Optional(), URL()])
+	website       = StringField("Website URL", [Optional(), URL()], filters = [lambda x: x or None])
-	issueTracker  = StringField("Issue Tracker URL", [Optional(), URL()])
+	issueTracker  = StringField("Issue Tracker URL", [Optional(), URL()], filters = [lambda x: x or None])
 	forums	      = IntegerField("Forum Topic ID", [Optional(), NumberRange(0,999999)])
 	submit	      = SubmitField("Save")
 
-@app.route("/packages/new/", methods=["GET", "POST"])
+@bp.route("/packages/new/", methods=["GET", "POST"])
-@app.route("/packages/<author>/<name>/edit/", methods=["GET", "POST"])
+@bp.route("/packages/<author>/<name>/edit/", methods=["GET", "POST"])
 @login_required
-def create_edit_package_page(author=None, name=None):
+def create_edit(author=None, name=None):
 	package = None
 	form = None
 	if author is None:
@@ -169,11 +204,11 @@ def create_edit_package_page(author=None, name=None):
 			author = User.query.filter_by(username=author).first()
 			if author is None:
 				flash("Unable to find that user", "error")
-				return redirect(url_for("create_edit_package_page"))
+				return redirect(url_for("packages.create_edit"))
 
 			if not author.checkPerm(current_user, Permission.CHANGE_AUTHOR):
 				flash("Permission denied", "error")
-				return redirect(url_for("create_edit_package_page"))
+				return redirect(url_for("packages.create_edit"))
 
 	else:
 		package = getPackageByInfo(author, name)
@@ -206,17 +241,26 @@ def create_edit_package_page(author=None, name=None):
 					Package.query.filter_by(name=form["name"].data, author_id=author.id).delete()
 				else:
 					flash("Package already exists!", "error")
-					return redirect(url_for("create_edit_package_page"))
+					return redirect(url_for("packages.create_edit"))
 
 			package = Package()
 			package.author = author
 			wasNew = True
+
+		elif package.approved and package.name != form.name.data and \
+				not package.checkPerm(current_user, Permission.CHANGE_NAME):
+			flash("Unable to change package name", "danger")
+			return redirect(url_for("packages.create_edit", author=author, name=name))
+
 		else:
 			triggerNotif(package.author, current_user,
 					"{} edited".format(package.title), package.getDetailsURL())
 
 		form.populate_obj(package) # copy to row
 
+		if package.type== PackageType.TXP:
+			package.license = package.media_license
+
 		mpackage_cache = {}
 		package.provides.clear()
 		mpackages = MetaPackage.SpecToList(form.provides_str.data, mpackage_cache)
@@ -244,11 +288,15 @@ def create_edit_package_page(author=None, name=None):
 
 		db.session.commit() # save
 
+		next_url = package.getDetailsURL()
 		if wasNew and package.repo is not None:
 			task = importRepoScreenshot.delay(package.id)
-			return redirect(url_for("check_task", id=task.id, r=package.getDetailsURL()))
+			next_url = url_for("tasks.check", id=task.id, r=next_url)
 
-		return redirect(package.getDetailsURL())
+		if wasNew and ("WTFPL" in package.license.name or "WTFPL" in package.media_license.name):
+			next_url = url_for("flatpage", path="help/wtfpl", r=next_url)
+
+		return redirect(next_url)
 
 	package_query = Package.query.filter_by(approved=True, soft_deleted=False)
 	if package is not None:
@@ -260,10 +308,10 @@ def create_edit_package_page(author=None, name=None):
 			packages=package_query.all(), \
 			mpackages=MetaPackage.query.order_by(db.asc(MetaPackage.name)).all())
 
-@app.route("/packages/<author>/<name>/approve/", methods=["POST"])
+@bp.route("/packages/<author>/<name>/approve/", methods=["POST"])
 @login_required
 @is_package_page
-def approve_package_page(package):
+def approve(package):
 	if not package.checkPerm(current_user, Permission.APPROVE_NEW):
 		flash("You don't have permission to do that.", "error")
 
@@ -284,25 +332,41 @@ def approve_package_page(package):
 	return redirect(package.getDetailsURL())
 
 
-@app.route("/packages/<author>/<name>/delete/", methods=["GET", "POST"])
+@bp.route("/packages/<author>/<name>/remove/", methods=["GET", "POST"])
 @login_required
 @is_package_page
-def delete_package_page(package):
+def remove(package):
 	if request.method == "GET":
-		return render_template("packages/delete.html", package=package)
+		return render_template("packages/remove.html", package=package)
 
-	if not package.checkPerm(current_user, Permission.DELETE_PACKAGE):
+	if "delete" in request.form:
-		flash("You don't have permission to do that.", "error")
+		if not package.checkPerm(current_user, Permission.DELETE_PACKAGE):
+			flash("You don't have permission to do that.", "error")
+			return redirect(package.getDetailsURL())
 
-	package.soft_deleted = True
+		package.soft_deleted = True
 
-	url = url_for("user_profile_page", username=package.author.username)
+		url = url_for("users.profile", username=package.author.username)
-	triggerNotif(package.author, current_user,
+		triggerNotif(package.author, current_user,
-			"{} deleted".format(package.title), url)
+				"{} deleted".format(package.title), url)
-	db.session.commit()
+		db.session.commit()
 
-	flash("Deleted package", "success")
+		flash("Deleted package", "success")
 
-	return redirect(url)
+		return redirect(url)
+	elif "unapprove" in request.form:
+		if not package.checkPerm(current_user, Permission.UNAPPROVE_PACKAGE):
+			flash("You don't have permission to do that.", "error")
+			return redirect(package.getDetailsURL())
 
-from . import todo, screenshots, releases
+		package.approved = False
+
+		triggerNotif(package.author, current_user,
+				"{} deleted".format(package.title), package.getDetailsURL())
+		db.session.commit()
+
+		flash("Unapproved package", "success")
+
+		return redirect(package.getDetailsURL())
+	else:
+		abort(400)

app/blueprints/packages/releases.py

@@ -0,0 +1,252 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_user import *
+
+from . import bp
+
+from app.rediscache import has_key, set_key, make_download_key
+from app.models import *
+from app.tasks.importtasks import makeVCSRelease, checkZipRelease
+from app.utils import *
+
+from celery import uuid
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+from wtforms.ext.sqlalchemy.fields import QuerySelectField
+
+
+def get_mt_releases(is_max):
+	query = MinetestRelease.query.order_by(db.asc(MinetestRelease.id))
+	if is_max:
+		query = query.limit(query.count() - 1)
+	else:
+		query = query.filter(MinetestRelease.name != "0.4.17")
+
+	return query
+
+
+class CreatePackageReleaseForm(FlaskForm):
+	title	   = StringField("Title", [InputRequired(), Length(1, 30)])
+	uploadOpt  = RadioField ("Method", choices=[("upload", "File Upload")], default="upload")
+	vcsLabel   = StringField("VCS Commit Hash, Branch, or Tag", default="master")
+	fileUpload = FileField("File Upload")
+	min_rel    = QuerySelectField("Minimum Minetest Version", [InputRequired()],
+			query_factory=lambda: get_mt_releases(False), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	max_rel    = QuerySelectField("Maximum Minetest Version", [InputRequired()],
+			query_factory=lambda: get_mt_releases(True), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	submit	   = SubmitField("Save")
+
+class EditPackageReleaseForm(FlaskForm):
+	title    = StringField("Title", [InputRequired(), Length(1, 30)])
+	url      = StringField("URL", [URL])
+	task_id  = StringField("Task ID", filters = [lambda x: x or None])
+	approved = BooleanField("Is Approved")
+	min_rel  = QuerySelectField("Minimum Minetest Version", [InputRequired()],
+			query_factory=lambda: get_mt_releases(False), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	max_rel  = QuerySelectField("Maximum Minetest Version", [InputRequired()],
+			query_factory=lambda: get_mt_releases(True), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	submit   = SubmitField("Save")
+
+@bp.route("/packages/<author>/<name>/releases/new/", methods=["GET", "POST"])
+@login_required
+@is_package_page
+def create_release(package):
+	if not package.checkPerm(current_user, Permission.MAKE_RELEASE):
+		return redirect(package.getDetailsURL())
+
+	# Initial form class from post data and default data
+	form = CreatePackageReleaseForm()
+	if package.repo is not None:
+		form["uploadOpt"].choices = [("vcs", "From Git Commit or Branch"), ("upload", "File Upload")]
+		if request.method != "POST":
+			form["uploadOpt"].data = "vcs"
+
+	if request.method == "POST" and form.validate():
+		if form["uploadOpt"].data == "vcs":
+			rel = PackageRelease()
+			rel.package = package
+			rel.title   = form["title"].data
+			rel.url     = ""
+			rel.task_id = uuid()
+			rel.min_rel = form["min_rel"].data.getActual()
+			rel.max_rel = form["max_rel"].data.getActual()
+			db.session.add(rel)
+			db.session.commit()
+
+			makeVCSRelease.apply_async((rel.id, form["vcsLabel"].data), task_id=rel.task_id)
+
+			msg = "{}: Release {} created".format(package.title, rel.title)
+			triggerNotif(package.author, current_user, msg, rel.getEditURL())
+			db.session.commit()
+
+			return redirect(url_for("tasks.check", id=rel.task_id, r=rel.getEditURL()))
+		else:
+			uploadedUrl, uploadedPath = doFileUpload(form.fileUpload.data, "zip", "a zip file")
+			if uploadedUrl is not None:
+				rel = PackageRelease()
+				rel.package = package
+				rel.title = form["title"].data
+				rel.url = uploadedUrl
+				rel.task_id = uuid()
+				rel.min_rel = form["min_rel"].data.getActual()
+				rel.max_rel = form["max_rel"].data.getActual()
+				db.session.add(rel)
+				db.session.commit()
+
+				checkZipRelease.apply_async((rel.id, uploadedPath), task_id=rel.task_id)
+
+				msg = "{}: Release {} created".format(package.title, rel.title)
+				triggerNotif(package.author, current_user, msg, rel.getEditURL())
+				db.session.commit()
+
+				return redirect(url_for("tasks.check", id=rel.task_id, r=rel.getEditURL()))
+
+	return render_template("packages/release_new.html", package=package, form=form)
+
+@bp.route("/packages/<author>/<name>/releases/<id>/download/")
+@is_package_page
+def download_release(package, id):
+	release = PackageRelease.query.get(id)
+	if release is None or release.package != package:
+		abort(404)
+
+	ip = request.headers.get("X-Forwarded-For") or request.remote_addr
+	if ip is not None:
+		key = make_download_key(ip, release.package)
+		if not has_key(key):
+			set_key(key, "true")
+
+			bonus = 1
+			if not package.getIsFOSS():
+				bonus *= 0.1
+
+			PackageRelease.query.filter_by(id=release.id).update({
+					"downloads": PackageRelease.downloads + 1
+				})
+
+			Package.query.filter_by(id=package.id).update({
+					"score": Package.score + bonus
+				})
+
+			db.session.commit()
+
+	return redirect(release.url, code=300)
+
+@bp.route("/packages/<author>/<name>/releases/<id>/", methods=["GET", "POST"])
+@login_required
+@is_package_page
+def edit_release(package, id):
+	release = PackageRelease.query.get(id)
+	if release is None or release.package != package:
+		abort(404)
+
+	clearNotifications(release.getEditURL())
+
+	canEdit	= package.checkPerm(current_user, Permission.MAKE_RELEASE)
+	canApprove = package.checkPerm(current_user, Permission.APPROVE_RELEASE)
+	if not (canEdit or canApprove):
+		return redirect(package.getDetailsURL())
+
+	# Initial form class from post data and default data
+	form = EditPackageReleaseForm(formdata=request.form, obj=release)
+
+	# HACK: fix bug in wtforms
+	if request.method == "GET":
+		form.approved.data = release.approved
+
+	if request.method == "POST" and form.validate():
+		wasApproved = release.approved
+		if canEdit:
+			release.title = form["title"].data
+			release.min_rel = form["min_rel"].data.getActual()
+			release.max_rel = form["max_rel"].data.getActual()
+
+		if package.checkPerm(current_user, Permission.CHANGE_RELEASE_URL):
+			release.url = form["url"].data
+			release.task_id = form["task_id"].data
+			if release.task_id is not None:
+				release.task_id = None
+
+		if canApprove:
+			release.approved = form["approved"].data
+		else:
+			release.approved = wasApproved
+
+		db.session.commit()
+		return redirect(package.getDetailsURL())
+
+	return render_template("packages/release_edit.html", package=package, release=release, form=form)
+
+
+
+class BulkReleaseForm(FlaskForm):
+	set_min = BooleanField("Set Min")
+	min_rel  = QuerySelectField("Minimum Minetest Version", [InputRequired()],
+			query_factory=lambda: get_mt_releases(False), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	set_max = BooleanField("Set Max")
+	max_rel  = QuerySelectField("Maximum Minetest Version", [InputRequired()],
+			query_factory=lambda: get_mt_releases(True), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	only_change_none = BooleanField("Only change values previously set as none")
+	submit   = SubmitField("Update")
+
+
+@bp.route("/packages/<author>/<name>/releases/bulk_change/", methods=["GET", "POST"])
+@login_required
+@is_package_page
+def bulk_change_release(package):
+	if not package.checkPerm(current_user, Permission.MAKE_RELEASE):
+		return redirect(package.getDetailsURL())
+
+	# Initial form class from post data and default data
+	form = BulkReleaseForm()
+
+	if request.method == "GET":
+		form.only_change_none.data = True
+	elif request.method == "POST" and form.validate():
+		only_change_none = form.only_change_none.data
+
+		for release in package.releases.all():
+			if form["set_min"].data and (not only_change_none or release.min_rel is None):
+				release.min_rel = form["min_rel"].data.getActual()
+			if form["set_max"].data and (not only_change_none or release.max_rel is None):
+				release.max_rel = form["max_rel"].data.getActual()
+
+		db.session.commit()
+
+		return redirect(package.getDetailsURL())
+
+	return render_template("packages/release_bulk_change.html", package=package, form=form)
+
+
+@bp.route("/packages/<author>/<name>/releases/<id>/delete/", methods=["POST"])
+@login_required
+@is_package_page
+def delete_release(package, id):
+	release = PackageRelease.query.get(id)
+	if release is None or release.package != package:
+		abort(404)
+
+	if not release.checkPerm(current_user, Permission.DELETE_RELEASE):
+		return redirect(release.getEditURL())
+
+	db.session.delete(release)
+	db.session.commit()
+
+	return redirect(package.getDetailsURL())

app/blueprints/packages/screenshots.py

@@ -17,9 +17,10 @@
 
 from flask import *
 from flask_user import *
-from app import app
-from app.models import *
 
+from . import bp
+
+from app.models import *
 from app.utils import *
 
 from flask_wtf import FlaskForm
@@ -39,23 +40,24 @@ class EditScreenshotForm(FlaskForm):
 	delete   = BooleanField("Delete")
 	submit   = SubmitField("Save")
 
-@app.route("/packages/<author>/<name>/screenshots/new/", methods=["GET", "POST"])
+@bp.route("/packages/<author>/<name>/screenshots/new/", methods=["GET", "POST"])
 @login_required
 @is_package_page
-def create_screenshot_page(package, id=None):
+def create_screenshot(package, id=None):
 	if not package.checkPerm(current_user, Permission.ADD_SCREENSHOTS):
 		return redirect(package.getDetailsURL())
 
 	# Initial form class from post data and default data
 	form = CreateScreenshotForm()
 	if request.method == "POST" and form.validate():
-		uploadedPath = doFileUpload(form.fileUpload.data, ["png", "jpg", "jpeg"],
+		uploadedUrl, uploadedPath = doFileUpload(form.fileUpload.data, "image",
 				"a PNG or JPG image file")
-		if uploadedPath is not None:
+		if uploadedUrl is not None:
 			ss = PackageScreenshot()
-			ss.package = package
+			ss.package  = package
-			ss.title   = form["title"].data
+			ss.title    = form["title"].data or "Untitled"
-			ss.url     = uploadedPath
+			ss.url      = uploadedUrl
+			ss.approved = package.checkPerm(current_user, Permission.APPROVE_SCREENSHOT)
 			db.session.add(ss)
 
 			msg = "{}: Screenshot added {}" \
@@ -66,10 +68,10 @@ def create_screenshot_page(package, id=None):
 
 	return render_template("packages/screenshot_new.html", package=package, form=form)
 
-@app.route("/packages/<author>/<name>/screenshots/<id>/edit/", methods=["GET", "POST"])
+@bp.route("/packages/<author>/<name>/screenshots/<id>/edit/", methods=["GET", "POST"])
 @login_required
 @is_package_page
-def edit_screenshot_page(package, id):
+def edit_screenshot(package, id):
 	screenshot = PackageScreenshot.query.get(id)
 	if screenshot is None or screenshot.package != package:
 		abort(404)
@@ -91,7 +93,7 @@ def edit_screenshot_page(package, id):
 			wasApproved = screenshot.approved
 
 			if canEdit:
-				screenshot.title = form["title"].data
+				screenshot.title = form["title"].data or "Untitled"
 
 			if canApprove:
 				screenshot.approved = form["approved"].data

app/blueprints/tasks/__init__.py

@@ -17,29 +17,29 @@
 
 from flask import *
 from flask_user import *
-from flask.ext import menu
+import flask_menu as menu
-from app import app, csrf
+from app import csrf
 from app.models import *
 from app.tasks import celery, TaskError
 from app.tasks.importtasks import getMeta
 from app.utils import shouldReturnJson
-# from celery.result import AsyncResult
-
 from app.utils import *
 
+bp = Blueprint("tasks", __name__)
+
 @csrf.exempt
-@app.route("/tasks/getmeta/new/", methods=["POST"])
+@bp.route("/tasks/getmeta/new/", methods=["POST"])
 @login_required
-def new_getmeta_page():
+def start_getmeta():
 	author = request.args.get("author")
 	author = current_user.forums_username if author is None else author
 	aresult = getMeta.delay(request.args.get("url"), author)
 	return jsonify({
-		"poll_url": url_for("check_task", id=aresult.id),
+		"poll_url": url_for("tasks.check", id=aresult.id),
 	})
 
-@app.route("/tasks/<id>/")
+@bp.route("/tasks/<id>/")
-def check_task(id):
+def check(id):
 	result = celery.AsyncResult(id)
 	status = result.status
 	traceback = result.traceback

app/blueprints/threads/__init__.py

@@ -16,23 +16,64 @@
 
 
 from flask import *
+
+bp = Blueprint("threads", __name__)
+
 from flask_user import *
-from app import app
 from app.models import *
 from app.utils import triggerNotif, clearNotifications
 
+import datetime
+
 from flask_wtf import FlaskForm
 from wtforms import *
 from wtforms.validators import *
 
-@app.route("/threads/")
+@bp.route("/threads/")
-def threads_page():
+def list_all():
-	threads = Thread.query.filter_by(private=False).all()
+	query = Thread.query
-	return render_template("threads/list.html", threads=threads)
+	if not Permission.SEE_THREAD.check(current_user):
+		query = query.filter_by(private=False)
+	return render_template("threads/list.html", threads=query.all())
 
-@app.route("/threads/<int:id>/", methods=["GET", "POST"])
+
-def thread_page(id):
+@bp.route("/threads/<int:id>/subscribe/", methods=["POST"])
-	clearNotifications(url_for("thread_page", id=id))
+@login_required
+def subscribe(id):
+	thread = Thread.query.get(id)
+	if thread is None or not thread.checkPerm(current_user, Permission.SEE_THREAD):
+		abort(404)
+
+	if current_user in thread.watchers:
+		flash("Already subscribed!", "success")
+	else:
+		flash("Subscribed to thread", "success")
+		thread.watchers.append(current_user)
+		db.session.commit()
+
+	return redirect(url_for("threads.view", id=id))
+
+
+@bp.route("/threads/<int:id>/unsubscribe/", methods=["POST"])
+@login_required
+def unsubscribe(id):
+	thread = Thread.query.get(id)
+	if thread is None or not thread.checkPerm(current_user, Permission.SEE_THREAD):
+		abort(404)
+
+	if current_user in thread.watchers:
+		flash("Unsubscribed!", "success")
+		thread.watchers.remove(current_user)
+		db.session.commit()
+	else:
+		flash("Not subscribed to thread", "success")
+
+	return redirect(url_for("threads.view", id=id))
+
+
+@bp.route("/threads/<int:id>/", methods=["GET", "POST"])
+def view(id):
+	clearNotifications(url_for("threads.view", id=id))
 
 	thread = Thread.query.get(id)
 	if thread is None or not thread.checkPerm(current_user, Permission.SEE_THREAD):
@@ -41,6 +82,13 @@ def thread_page(id):
 	if current_user.is_authenticated and request.method == "POST":
 		comment = request.form["comment"]
 
+		if not current_user.canCommentRL():
+			flash("Please wait before commenting again", "danger")
+			if package:
+				return redirect(package.getDetailsURL())
+			else:
+				return redirect(url_for("homepage.home"))
+
 		if len(comment) <= 500 and len(comment) > 3:
 			reply = ThreadReply()
 			reply.author = current_user
@@ -60,11 +108,11 @@ def thread_page(id):
 
 			for user in thread.watchers:
 				if user != current_user:
-					triggerNotif(user, current_user, msg, url_for("thread_page", id=thread.id))
+					triggerNotif(user, current_user, msg, url_for("threads.view", id=thread.id))
 
 			db.session.commit()
 
-			return redirect(url_for("thread_page", id=id))
+			return redirect(url_for("threads.view", id=id))
 
 		else:
 			flash("Comment needs to be between 3 and 500 characters.")
@@ -78,9 +126,9 @@ class ThreadForm(FlaskForm):
 	private = BooleanField("Private")
 	submit  = SubmitField("Open Thread")
 
-@app.route("/threads/new/", methods=["GET", "POST"])
+@bp.route("/threads/new/", methods=["GET", "POST"])
 @login_required
-def new_thread_page():
+def new():
 	form = ThreadForm(formdata=request.form)
 
 	package = None
@@ -89,27 +137,33 @@ def new_thread_page():
 		if package is None:
 			flash("Unable to find that package!", "error")
 
-	# Don't allow making threads on approved packages for now
+	# Don't allow making orphan threads on approved packages for now
-	if package is None or package.approved:
+	if package is None:
 		abort(403)
 
 	def_is_private   = request.args.get("private") or False
-	if not package.approved:
+	if package is None:
 		def_is_private = True
-	allow_change     = package.approved
+	allow_change     = package and package.approved
-	is_review_thread = package is not None and not package.approved
+	is_review_thread = package and not package.approved
 
 	# Check that user can make the thread
-	if is_review_thread and not (package.author == current_user or \
+	if not package.checkPerm(current_user, Permission.CREATE_THREAD):
-			package.checkPerm(current_user, Permission.APPROVE_NEW)):
 		flash("Unable to create thread!", "error")
-		return redirect(url_for("home_page"))
+		return redirect(url_for("homepage.home"))
 
 	# Only allow creating one thread when not approved
 	elif is_review_thread and package.review_thread is not None:
 		flash("A review thread already exists!", "error")
-		if request.method == "GET":
+		return redirect(url_for("threads.view", id=package.review_thread.id))
-			return redirect(url_for("thread_page", id=package.review_thread.id))
+
+	elif not current_user.canOpenThreadRL():
+		flash("Please wait before opening another thread", "danger")
+
+		if package:
+			return redirect(package.getDetailsURL())
+		else:
+			return redirect(url_for("homepage.home"))
 
 	# Set default values
 	elif request.method == "GET":
@@ -142,13 +196,19 @@ def new_thread_page():
 		if is_review_thread:
 			package.review_thread = thread
 
+		notif_msg = None
 		if package is not None:
-			triggerNotif(package.author, current_user,
+			notif_msg = "New thread '{}' on package {}".format(thread.title, package.title)
-					"New thread '{}' on package {}".format(thread.title, package.title), url_for("thread_page", id=thread.id))
+			triggerNotif(package.author, current_user, notif_msg, url_for("threads.view", id=thread.id))
+		else:
+			notif_msg = "New thread '{}'".format(thread.title)
+
+		for user in User.query.filter(User.rank >= UserRank.EDITOR).all():
+			triggerNotif(user, current_user, notif_msg, url_for("threads.view", id=thread.id))
 
 		db.session.commit()
 
-		return redirect(url_for("thread_page", id=thread.id))
+		return redirect(url_for("threads.view", id=thread.id))
 
 
-	return render_template("threads/new.html", form=form, allow_private_change=allow_change)
+	return render_template("threads/new.html", form=form, allow_private_change=allow_change, package=package)

app/blueprints/thumbnails/__init__.py

@@ -0,0 +1,79 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+
+bp = Blueprint("thumbnails", __name__)
+
+import os
+from PIL import Image
+
+ALLOWED_RESOLUTIONS=[(100,67), (270,180), (350,233)]
+
+def mkdir(path):
+	assert path != "" and path is not None
+	if not os.path.isdir(path):
+		os.mkdir(path)
+
+
+def resize_and_crop(img_path, modified_path, size):
+	img = Image.open(img_path)
+
+	# Get current and desired ratio for the images
+	img_ratio = img.size[0] / float(img.size[1])
+	ratio = size[0] / float(size[1])
+
+	# Is more portrait than target, scale and crop
+	if ratio > img_ratio:
+		img = img.resize((int(size[0]), int(size[0] * img.size[1] / img.size[0])),
+				Image.BICUBIC)
+		box = (0, (img.size[1] - size[1]) / 2, img.size[0], (img.size[1] + size[1]) / 2)
+		img = img.crop(box)
+
+	# Is more landscape than target, scale and crop
+	elif ratio < img_ratio:
+		img = img.resize((int(size[1] * img.size[0] / img.size[1]), int(size[1])),
+				Image.BICUBIC)
+		box = ((img.size[0] - size[0]) / 2, 0, (img.size[0] + size[0]) / 2, img.size[1])
+		img = img.crop(box)
+
+	# Is exactly the same ratio as target
+	else:
+		img = img.resize(size, Image.BICUBIC)
+
+	img.save(modified_path)
+
+
+@bp.route("/thumbnails/<int:level>/<img>")
+def make_thumbnail(img, level):
+	if level > len(ALLOWED_RESOLUTIONS) or level <= 0:
+		abort(403)
+
+	w, h = ALLOWED_RESOLUTIONS[level - 1]
+
+	upload_dir = current_app.config["UPLOAD_DIR"]
+	thumbnail_dir = current_app.config["THUMBNAIL_DIR"]
+	mkdir(thumbnail_dir)
+
+	output_dir = os.path.join(thumbnail_dir, str(level))
+	mkdir(output_dir)
+
+	cache_filepath  = os.path.join(output_dir, img)
+	source_filepath = os.path.join(upload_dir, img)
+
+	resize_and_crop(source_filepath, cache_filepath, (w, h))
+	return send_file(cache_filepath)

app/blueprints/todo/__init__.py

@@ -0,0 +1,102 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+from flask import *
+from flask_user import *
+import flask_menu as menu
+from app.models import *
+from app.querybuilder import QueryBuilder
+from app.utils import get_int_or_abort
+
+bp = Blueprint("todo", __name__)
+
+@bp.route("/todo/", methods=["GET", "POST"])
+@login_required
+def view():
+	canApproveNew = Permission.APPROVE_NEW.check(current_user)
+	canApproveRel = Permission.APPROVE_RELEASE.check(current_user)
+	canApproveScn = Permission.APPROVE_SCREENSHOT.check(current_user)
+
+	packages = None
+	if canApproveNew:
+		packages = Package.query.filter_by(approved=False, soft_deleted=False).order_by(db.desc(Package.created_at)).all()
+
+	releases = None
+	if canApproveRel:
+		releases = PackageRelease.query.filter_by(approved=False).all()
+
+	screenshots = None
+	if canApproveScn:
+		screenshots = PackageScreenshot.query.filter_by(approved=False).all()
+
+	if not canApproveNew and not canApproveRel and not canApproveScn:
+		abort(403)
+
+	if request.method == "POST":
+		if request.form["action"] == "screenshots_approve_all":
+			if not canApproveScn:
+				abort(403)
+
+			PackageScreenshot.query.update({ "approved": True })
+			db.session.commit()
+			return redirect(url_for("todo.view"))
+		else:
+			abort(400)
+
+	topic_query = ForumTopic.query \
+			.filter_by(discarded=False)
+
+	total_topics = topic_query.count()
+	topics_to_add = topic_query \
+			.filter(~ db.exists().where(Package.forums==ForumTopic.topic_id)) \
+			.count()
+
+	return render_template("todo/list.html", title="Reports and Work Queue",
+		packages=packages, releases=releases, screenshots=screenshots,
+		canApproveNew=canApproveNew, canApproveRel=canApproveRel, canApproveScn=canApproveScn,
+		topics_to_add=topics_to_add, total_topics=total_topics)
+
+
+@bp.route("/todo/topics/")
+@login_required
+def topics():
+	qb    = QueryBuilder(request.args)
+	qb.setSortIfNone("date")
+	query = qb.buildTopicQuery()
+
+	tmp_q = ForumTopic.query
+	if not qb.show_discarded:
+		tmp_q = tmp_q.filter_by(discarded=False)
+	total = tmp_q.count()
+	topic_count = query.count()
+
+	page  = get_int_or_abort(request.args.get("page"), 1)
+	num   = get_int_or_abort(request.args.get("n"), 100)
+	if num > 100 and not current_user.rank.atLeast(UserRank.EDITOR):
+		num = 100
+
+	query = query.paginate(page, num, True)
+	next_url = url_for("todo.topics", page=query.next_num, query=qb.search, \
+	 	show_discarded=qb.show_discarded, n=num, sort=qb.order_by) \
+			if query.has_next else None
+	prev_url = url_for("todo.topics", page=query.prev_num, query=qb.search, \
+	 	show_discarded=qb.show_discarded, n=num, sort=qb.order_by) \
+			if query.has_prev else None
+
+	return render_template("todo/topics.html", topics=query.items, total=total, \
+			topic_count=topic_count, query=qb.search, show_discarded=qb.show_discarded, \
+			next_url=next_url, prev_url=prev_url, page=page, page_max=query.pages, \
+			n=num, sort_by=qb.order_by)

app/blueprints/users/__init__.py

@@ -0,0 +1,5 @@
+from flask import Blueprint
+
+bp = Blueprint("users", __name__)
+
+from . import githublogin, profile

app/blueprints/users/githublogin.py

@@ -21,15 +21,16 @@ from flask_login import login_user, logout_user
 from sqlalchemy import func
 import flask_menu as menu
 from flask_github import GitHub
-from app import app, github
+from . import bp
+from app import github
 from app.models import *
 from app.utils import loginUser
 
-@app.route("/user/github/start/")
+@bp.route("/user/github/start/")
-def github_signin_page():
+def github_signin():
 	return github.authorize("")
 
-@app.route("/user/github/callback/")
+@bp.route("/user/github/callback/")
 @github.authorized_handler
 def github_authorized(oauth_token):
 	next_url = request.args.get("next")
@@ -51,23 +52,23 @@ def github_authorized(oauth_token):
 	if current_user and current_user.is_authenticated:
 		if userByGithub is None:
 			current_user.github_username = username
-			db.session.add(auth)
 			db.session.commit()
-			return redirect(url_for("gitAccount", id=auth.id))
+			flash("Linked github to account", "success")
+			return redirect(url_for("homepage.home"))
 		else:
 			flash("Github account is already associated with another user", "danger")
-			return redirect(url_for("home_page"))
+			return redirect(url_for("homepage.home"))
 
 	# If not logged in, log in
 	else:
 		if userByGithub is None:
 			flash("Unable to find an account for that Github user", "error")
-			return redirect(url_for("user_claim_page"))
+			return redirect(url_for("users.claim"))
 		elif loginUser(userByGithub):
-			if current_user.password is None:
+			if not current_user.hasPassword():
-				return redirect(next_url or url_for("set_password_page", optional=True))
+				return redirect(next_url or url_for("users.set_password", optional=True))
 			else:
-				return redirect(next_url or url_for("home_page"))
+				return redirect(next_url or url_for("homepage.home"))
 		else:
 			flash("Authorization failed [err=gh-login-failed]", "danger")
 			return redirect(url_for("user.login"))

app/blueprints/users/profile.py

@@ -18,34 +18,35 @@
 from flask import *
 from flask_user import *
 from flask_login import login_user, logout_user
-from flask.ext import menu
+from app import markdown
-from app import app
+from . import bp
 from app.models import *
 from flask_wtf import FlaskForm
-from flask_user.forms import RegisterForm
 from wtforms import *
 from wtforms.validators import *
-from app.utils import rank_required, randomString, loginUser
+from app.utils import randomString, loginUser, rank_required
 from app.tasks.forumtasks import checkForumAccount
-from app.tasks.emails import sendVerifyEmail
+from app.tasks.emails import sendVerifyEmail, sendEmailRaw
 from app.tasks.phpbbparser import getProfile
 
 # Define the User profile form
 class UserProfileForm(FlaskForm):
 	display_name = StringField("Display name", [Optional(), Length(2, 20)])
-	email = StringField("Email", [Optional(), Email()])
+	email = StringField("Email", [Optional(), Email()], filters = [lambda x: x or None])
+	website_url = StringField("Website URL", [Optional(), URL()], filters = [lambda x: x or None])
+	donate_url = StringField("Donation URL", [Optional(), URL()], filters = [lambda x: x or None])
 	rank = SelectField("Rank", [Optional()], choices=UserRank.choices(), coerce=UserRank.coerce, default=UserRank.NEW_MEMBER)
 	submit = SubmitField("Save")
 
-@app.route("/users/", methods=["GET"])
+
-@login_required
+@bp.route("/users/", methods=["GET"])
-def user_list_page():
+def list_all():
 	users = User.query.order_by(db.desc(User.rank), db.asc(User.display_name)).all()
 	return render_template("users/list.html", users=users)
 
 
-@app.route("/users/<username>/", methods=["GET", "POST"])
+@bp.route("/users/<username>/", methods=["GET", "POST"])
-def user_profile_page(username):
+def profile(username):
 	user = User.query.filter_by(username=username).first()
 	if not user:
 		abort(404)
@@ -62,6 +63,8 @@ def user_profile_page(username):
 			# Copy form fields to user_profile fields
 			if user.checkPerm(current_user, Permission.CHANGE_DNAME):
 				user.display_name = form["display_name"].data
+				user.website_url  = form["website_url"].data
+				user.donate_url   = form["donate_url"].data
 
 			if user.checkPerm(current_user, Permission.CHANGE_RANK):
 				newRank = form["rank"].data
@@ -76,20 +79,20 @@ def user_profile_page(username):
 					token = randomString(32)
 
 					ver = UserEmailVerification()
-					ver.user = user
+					ver.user  = user
 					ver.token = token
 					ver.email = newEmail
 					db.session.add(ver)
 					db.session.commit()
 
 					task = sendVerifyEmail.delay(newEmail, token)
-					return redirect(url_for("check_task", id=task.id, r=url_for("user_profile_page", username=username)))
+					return redirect(url_for("tasks.check", id=task.id, r=url_for("users.profile", username=username)))
 
 			# Save user_profile
 			db.session.commit()
 
 			# Redirect to home page
-			return redirect(url_for("user_profile_page", username=username))
+			return redirect(url_for("users.profile", username=username))
 
 	packages = user.packages.filter_by(soft_deleted=False)
 	if not current_user.is_authenticated or (user != current_user and not current_user.canAccessTodoList()):
@@ -98,26 +101,76 @@ def user_profile_page(username):
 
 	topics_to_add = None
 	if current_user == user or user.checkPerm(current_user, Permission.CHANGE_AUTHOR):
-		topics_to_add = KrockForumTopic.query \
+		topics_to_add = ForumTopic.query \
 					.filter_by(author_id=user.id) \
-					.filter(~ db.exists().where(Package.forums==KrockForumTopic.topic_id)) \
+					.filter(~ db.exists().where(Package.forums==ForumTopic.topic_id)) \
-					.order_by(db.asc(KrockForumTopic.name), db.asc(KrockForumTopic.title)) \
+					.order_by(db.asc(ForumTopic.name), db.asc(ForumTopic.title)) \
 					.all()
 
 	# Process GET or invalid POST
-	return render_template("users/user_profile_page.html",
+	return render_template("users/profile.html",
 			user=user, form=form, packages=packages, topics_to_add=topics_to_add)
 
+
+@bp.route("/users/<username>/check/", methods=["POST"])
+@login_required
+def user_check(username):
+	user = User.query.filter_by(username=username).first()
+	if user is None:
+		abort(404)
+
+	if current_user != user and not current_user.rank.atLeast(UserRank.MODERATOR):
+		abort(403)
+
+	if user.forums_username is None:
+		abort(404)
+
+	task = checkForumAccount.delay(user.forums_username)
+	next_url = url_for("users.profile", username=username)
+
+	return redirect(url_for("tasks.check", id=task.id, r=next_url))
+
+
+class SendEmailForm(FlaskForm):
+	subject = StringField("Subject", [InputRequired(), Length(1, 300)])
+	text    = TextAreaField("Message", [InputRequired()])
+	submit  = SubmitField("Send")
+
+
+@bp.route("/users/<username>/email/", methods=["GET", "POST"])
+@rank_required(UserRank.MODERATOR)
+def send_email(username):
+	user = User.query.filter_by(username=username).first()
+	if user is None:
+		abort(404)
+
+	next_url = url_for("users.profile", username=user.username)
+
+	if user.email is None:
+		flash("User has no email address!", "error")
+		return redirect(next_url)
+
+	form = SendEmailForm(request.form)
+	if form.validate_on_submit():
+		text = form.text.data
+		html = markdown(text)
+		task = sendEmailRaw.delay([user.email], form.subject.data, text, html)
+		return redirect(url_for("tasks.check", id=task.id, r=next_url))
+
+	return render_template("users/send_email.html", form=form)
+
+
+
 class SetPasswordForm(FlaskForm):
 	email = StringField("Email", [Optional(), Email()])
-	password = PasswordField("New password", [InputRequired(), Length(2, 20)])
+	password = PasswordField("New password", [InputRequired(), Length(2, 100)])
-	password2 = PasswordField("Verify password", [InputRequired(), Length(2, 20)])
+	password2 = PasswordField("Verify password", [InputRequired(), Length(2, 100)])
 	submit = SubmitField("Save")
 
-@app.route("/user/set-password/", methods=["GET", "POST"])
+@bp.route("/user/set-password/", methods=["GET", "POST"])
 @login_required
-def set_password_page():
+def set_password():
-	if current_user.password is not None:
+	if current_user.hasPassword():
 		return redirect(url_for("user.change_password"))
 
 	form = SetPasswordForm(request.form)
@@ -132,10 +185,11 @@ def set_password_page():
 			hashed_password = user_manager.hash_password(form.password.data)
 
 			# Change password
-			user_manager.update_password(current_user, hashed_password)
+			current_user.password = hashed_password
+			db.session.commit()
 
 			# Send 'password_changed' email
-			if user_manager.enable_email and user_manager.send_password_changed_email and current_user.email:
+			if user_manager.USER_ENABLE_EMAIL and current_user.email:
 				emails.send_password_changed_email(current_user)
 
 			# Send password_changed signal
@@ -156,17 +210,17 @@ def set_password_page():
 				db.session.commit()
 
 				task = sendVerifyEmail.delay(newEmail, token)
-				return redirect(url_for("check_task", id=task.id, r=url_for("user_profile_page", username=current_user.username)))
+				return redirect(url_for("tasks.check", id=task.id, r=url_for("users.profile", username=current_user.username)))
 			else:
-				return redirect(url_for("user_profile_page", username=current_user.username))
+				return redirect(url_for("user.login"))
 		else:
 			flash("Passwords do not match", "error")
 
 	return render_template("users/set_password.html", form=form, optional=request.args.get("optional"))
 
 
-@app.route("/user/claim/", methods=["GET", "POST"])
+@bp.route("/user/claim/", methods=["GET", "POST"])
-def user_claim_page():
+def claim():
 	username = request.args.get("username")
 	if username is None:
 		username = ""
@@ -175,16 +229,16 @@ def user_claim_page():
 		user = User.query.filter_by(forums_username=username).first()
 		if user and user.rank.atLeast(UserRank.NEW_MEMBER):
 			flash("User has already been claimed", "error")
-			return redirect(url_for("user_claim_page"))
+			return redirect(url_for("users.claim"))
 		elif user is None and method == "github":
 			flash("Unable to get Github username for user", "error")
-			return redirect(url_for("user_claim_page"))
+			return redirect(url_for("users.claim"))
 		elif user is None:
 			flash("Unable to find that user", "error")
-			return redirect(url_for("user_claim_page"))
+			return redirect(url_for("users.claim"))
 
 		if user is not None and method == "github":
-			return redirect(url_for("github_signin_page"))
+			return redirect(url_for("users.github_signin"))
 
 	token = None
 	if "forum_token" in session:
@@ -201,12 +255,12 @@ def user_claim_page():
 			flash("Invalid username", "error")
 		elif ctype == "github":
 			task = checkForumAccount.delay(username)
-			return redirect(url_for("check_task", id=task.id, r=url_for("user_claim_page", username=username, method="github")))
+			return redirect(url_for("tasks.check", id=task.id, r=url_for("users.claim", username=username, method="github")))
 		elif ctype == "forum":
 			user = User.query.filter_by(forums_username=username).first()
 			if user is not None and user.rank.atLeast(UserRank.NEW_MEMBER):
 				flash("That user has already been claimed!", "error")
-				return redirect(url_for("user_claim_page"))
+				return redirect(url_for("users.claim"))
 
 			# Get signature
 			sig = None
@@ -215,7 +269,7 @@ def user_claim_page():
 				sig = profile.signature
 			except IOError:
 				flash("Unable to get forum signature - does the user exist?", "error")
-				return redirect(url_for("user_claim_page", username=username))
+				return redirect(url_for("users.claim", username=username))
 
 			# Look for key
 			if token in sig:
@@ -226,21 +280,21 @@ def user_claim_page():
 					db.session.commit()
 
 				if loginUser(user):
-					return redirect(url_for("set_password_page"))
+					return redirect(url_for("users.set_password"))
 				else:
 					flash("Unable to login as user", "error")
-					return redirect(url_for("user_claim_page", username=username))
+					return redirect(url_for("users.claim", username=username))
 
 			else:
 				flash("Could not find the key in your signature!", "error")
-				return redirect(url_for("user_claim_page", username=username))
+				return redirect(url_for("users.claim", username=username))
 		else:
 			flash("Unknown claim type", "error")
 
 	return render_template("users/claim.html", username=username, key=token)
 
-@app.route("/users/verify/")
+@bp.route("/users/verify/")
-def verify_email_page():
+def verify_email():
 	token = request.args.get("token")
 	ver = UserEmailVerification.query.filter_by(token=token).first()
 	if ver is None:
@@ -251,6 +305,6 @@ def verify_email_page():
 		db.session.commit()
 
 	if current_user.is_authenticated:
-		return redirect(url_for("user_profile_page", username=current_user.username))
+		return redirect(url_for("users.profile", username=current_user.username))
 	else:
-		return redirect(url_for("home_page"))
+		return redirect(url_for("homepage.home"))

app/default_data.py

@@ -1,43 +1,65 @@
-# Content DB
+from .models import *
-# Copyright (C) 2018  rubenwardy
+from .utils import make_flask_user_password
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-import os, sys, datetime
+def populate(session):
+	admin_user = User("rubenwardy")
+	admin_user.active = True
+	admin_user.password = make_flask_user_password("tuckfrump")
+	admin_user.github_username = "rubenwardy"
+	admin_user.forums_username = "rubenwardy"
+	admin_user.rank = UserRank.ADMIN
+	session.add(admin_user)
 
-if not "FLASK_CONFIG" in os.environ:
+	session.add(MinetestRelease("None", 0))
-	os.environ["FLASK_CONFIG"] = "../config.cfg"
+	session.add(MinetestRelease("0.4.16/17", 32))
+	session.add(MinetestRelease("5.0", 37))
+	session.add(MinetestRelease("5.1", 38))
 
-test_data = len(sys.argv) >= 2 and sys.argv[1].strip() == "-t"
+	tags = {}
+	for tag in ["Inventory", "Mapgen", "Building", \
+			"Mobs and NPCs", "Tools", "Player effects", \
+			"Environment", "Transport", "Maintenance", "Plants and farming", \
+			"PvP", "PvE", "Survival", "Creative", "Puzzle", "Multiplayer", "Singleplayer"]:
+		row = Tag(tag)
+		tags[row.name] = row
+		session.add(row)
 
-from app.models import *
+	licenses = {}
+	for license in ["GPLv2.1", "GPLv3", "LGPLv2.1", "LGPLv3", "AGPLv2.1", "AGPLv3",
+					"Apache", "BSD 3-Clause", "BSD 2-Clause", "CC0", "CC-BY-SA",
+					"CC-BY", "MIT", "ZLib", "Other (Free)"]:
+		row = License(license)
+		licenses[row.name] = row
+		session.add(row)
+
+	for license in ["CC-BY-NC-SA", "Other (Non-free)"]:
+		row = License(license, False)
+		licenses[row.name] = row
+		session.add(row)
+
+
+def populate_test_data(session):
+	licenses = { x.name : x for x in License.query.all() }
+	tags = { x.name : x for x in Tag.query.all() }
+	admin_user = User.query.filter_by(rank=UserRank.ADMIN).first()
+	v4 = MinetestRelease.query.filter_by(protocol=32).first()
+	v50 = MinetestRelease.query.filter_by(protocol=37).first()
+	v51 = MinetestRelease.query.filter_by(protocol=38).first()
 
-def defineDummyData(licenses, tags, ruben):
 	ez = User("Shara")
 	ez.github_username = "Ezhh"
 	ez.forums_username = "Shara"
 	ez.rank = UserRank.EDITOR
-	db.session.add(ez)
+	session.add(ez)
 
-	not1 = Notification(ruben, ez, "Awards approved", "/packages/rubenwardy/awards/")
+	not1 = Notification(admin_user, ez, "Awards approved", "/packages/rubenwardy/awards/")
-	db.session.add(not1)
+	session.add(not1)
 
 	jeija = User("Jeija")
 	jeija.github_username = "Jeija"
 	jeija.forums_username = "Jeija"
-	db.session.add(jeija)
+	session.add(jeija)
 
 
 	mod = Package()
@@ -45,36 +67,38 @@ def defineDummyData(licenses, tags, ruben):
 	mod.name = "alpha"
 	mod.title = "Alpha Test"
 	mod.license = licenses["MIT"]
+	mod.media_license = licenses["MIT"]
 	mod.type = PackageType.MOD
-	mod.author = ruben
+	mod.author = admin_user
 	mod.tags.append(tags["mapgen"])
 	mod.tags.append(tags["environment"])
 	mod.repo = "https://github.com/ezhh/other_worlds"
 	mod.issueTracker = "https://github.com/ezhh/other_worlds/issues"
 	mod.forums = 16015
-	mod.shortDesc = "The content library should not be used yet as it is still in alpha"
+	mod.short_desc = "The content library should not be used yet as it is still in alpha"
 	mod.desc = "This is the long desc"
-	db.session.add(mod)
+	session.add(mod)
 
 	rel = PackageRelease()
 	rel.package = mod
 	rel.title = "v1.0.0"
 	rel.url = "https://github.com/ezhh/handholds/archive/master.zip"
 	rel.approved = True
-	db.session.add(rel)
+	session.add(rel)
 
 	mod1 = Package()
 	mod1.approved = True
 	mod1.name = "awards"
 	mod1.title = "Awards"
 	mod1.license = licenses["LGPLv2.1"]
+	mod1.media_license = licenses["MIT"]
 	mod1.type = PackageType.MOD
-	mod1.author = ruben
+	mod1.author = admin_user
 	mod1.tags.append(tags["player_effects"])
 	mod1.repo = "https://github.com/rubenwardy/awards"
 	mod1.issueTracker = "https://github.com/rubenwardy/awards/issues"
 	mod1.forums = 4870
-	mod1.shortDesc = "Adds achievements and an API to register new ones."
+	mod1.short_desc = "Adds achievements and an API to register new ones."
 	mod1.desc = """
 Majority of awards are back ported from Calinou's old fork in Carbone, under same license.
 
@@ -93,10 +117,11 @@ awards.register_achievement("award_mesefind",{
 
 	rel = PackageRelease()
 	rel.package = mod1
+	rel.min_rel = v51
 	rel.title = "v1.0.0"
 	rel.url = "https://github.com/rubenwardy/awards/archive/master.zip"
 	rel.approved = True
-	db.session.add(rel)
+	session.add(rel)
 
 	mod2 = Package()
 	mod2.approved = True
@@ -105,22 +130,13 @@ awards.register_achievement("award_mesefind",{
 	mod2.tags.append(tags["tools"])
 	mod2.type = PackageType.MOD
 	mod2.license = licenses["LGPLv3"]
+	mod2.media_license = licenses["MIT"]
 	mod2.author = jeija
 	mod2.repo = "https://github.com/minetest-mods/mesecons/"
 	mod2.issueTracker = "https://github.com/minetest-mods/mesecons/issues"
 	mod2.forums = 628
-	mod2.shortDesc = "Mesecons adds everything digital, from all kinds of sensors, switches, solar panels, detectors, pistons, lamps, sound blocks to advanced digital circuitry like logic gates and programmable blocks."
+	mod2.short_desc = "Mesecons adds everything digital, from all kinds of sensors, switches, solar panels, detectors, pistons, lamps, sound blocks to advanced digital circuitry like logic gates and programmable blocks."
 	mod2.desc = """
-    ########################################################################
-    ##  __    __   _____   _____   _____   _____   _____   _   _   _____  ##
-    ## |  \  /  | |  ___| |  ___| |  ___| |  ___| |  _  | | \ | | |  ___| ##
-    ## |   \/   | | |___  | |___  | |___  | |     | | | | |  \| | | |___  ##
-    ## | |\__/| | |  ___| |___  | |  ___| | |     | | | | |     | |___  | ##
-    ## | |    | | | |___   ___| | | |___  | |___  | |_| | | |\  |  ___| | ##
-    ## |_|    |_| |_____| |_____| |_____| |_____| |_____| |_| \_| |_____| ##
-    ##                                                                    ##
-    ########################################################################
-
 MESECONS by Jeija and contributors
 
 Mezzee-what?
@@ -193,36 +209,39 @@ No warranty is provided, express or implied, for any part of the project.
 
 """
 
-	db.session.add(mod1)
+	session.add(mod1)
-	db.session.add(mod2)
+	session.add(mod2)
 
 	mod = Package()
 	mod.approved = True
 	mod.name = "handholds"
 	mod.title = "Handholds"
 	mod.license = licenses["MIT"]
+	mod.media_license = licenses["MIT"]
 	mod.type = PackageType.MOD
 	mod.author = ez
 	mod.tags.append(tags["player_effects"])
 	mod.repo = "https://github.com/ezhh/handholds"
 	mod.issueTracker = "https://github.com/ezhh/handholds/issues"
 	mod.forums = 17069
-	mod.shortDesc = "Adds hand holds and climbing thingies"
+	mod.short_desc = "Adds hand holds and climbing thingies"
 	mod.desc = "This is the long desc"
-	db.session.add(mod)
+	session.add(mod)
 
 	rel = PackageRelease()
 	rel.package = mod
 	rel.title = "v1.0.0"
+	rel.max_rel = v4
 	rel.url = "https://github.com/ezhh/handholds/archive/master.zip"
 	rel.approved = True
-	db.session.add(rel)
+	session.add(rel)
 
 	mod = Package()
 	mod.approved = True
 	mod.name = "other_worlds"
 	mod.title = "Other Worlds"
 	mod.license = licenses["MIT"]
+	mod.media_license = licenses["MIT"]
 	mod.type = PackageType.MOD
 	mod.author = ez
 	mod.tags.append(tags["mapgen"])
@@ -230,41 +249,42 @@ No warranty is provided, express or implied, for any part of the project.
 	mod.repo = "https://github.com/ezhh/other_worlds"
 	mod.issueTracker = "https://github.com/ezhh/other_worlds/issues"
 	mod.forums = 16015
-	mod.shortDesc = "Adds space with asteroids and comets"
+	mod.short_desc = "Adds space with asteroids and comets"
 	mod.desc = "This is the long desc"
-	db.session.add(mod)
+	session.add(mod)
 
 	mod = Package()
 	mod.approved = True
 	mod.name = "food"
 	mod.title = "Food"
 	mod.license = licenses["LGPLv2.1"]
+	mod.media_license = licenses["MIT"]
 	mod.type = PackageType.MOD
-	mod.author = ruben
+	mod.author = admin_user
 	mod.tags.append(tags["player_effects"])
 	mod.repo = "https://github.com/rubenwardy/food/"
 	mod.issueTracker = "https://github.com/rubenwardy/food/issues/"
 	mod.forums = 2960
-	mod.shortDesc = "Adds lots of food and an API to manage ingredients"
+	mod.short_desc = "Adds lots of food and an API to manage ingredients"
 	mod.desc = "This is the long desc"
-	food = mod
+	session.add(mod)
-	db.session.add(mod)
 
 	mod = Package()
 	mod.approved = True
 	mod.name = "food_sweet"
 	mod.title = "Sweet Foods"
 	mod.license = licenses["CC0"]
+	mod.media_license = licenses["MIT"]
 	mod.type = PackageType.MOD
-	mod.author = ruben
+	mod.author = admin_user
 	mod.tags.append(tags["player_effects"])
 	mod.repo = "https://github.com/rubenwardy/food_sweet/"
 	mod.issueTracker = "https://github.com/rubenwardy/food_sweet/issues/"
 	mod.forums = 9039
-	mod.shortDesc = "Adds sweet food"
+	mod.short_desc = "Adds sweet food"
 	mod.desc = "This is the long desc"
 	food_sweet = mod
-	db.session.add(mod)
+	session.add(mod)
 
 	game1 = Package()
 	game1.approved = True
@@ -272,28 +292,29 @@ No warranty is provided, express or implied, for any part of the project.
 	game1.title = "Capture The Flag"
 	game1.type = PackageType.GAME
 	game1.license = licenses["LGPLv2.1"]
-	game1.author = ruben
+	game1.media_license = licenses["MIT"]
+	game1.author = admin_user
 	game1.tags.append(tags["pvp"])
 	game1.tags.append(tags["survival"])
 	game1.tags.append(tags["multiplayer"])
 	game1.repo = "https://github.com/rubenwardy/capturetheflag"
 	game1.issueTracker = "https://github.com/rubenwardy/capturetheflag/issues"
 	game1.forums = 12835
-	game1.shortDesc = "Two teams battle to snatch and return the enemy's flag, before the enemy takes their own!"
+	game1.short_desc = "Two teams battle to snatch and return the enemy's flag, before the enemy takes their own!"
 	game1.desc = """
 As seen on the Capture the Flag server (minetest.rubenwardy.com:30000)
 
 Uses the CTF PvP Engine.
 """
 
-	db.session.add(game1)
+	session.add(game1)
 
 	rel = PackageRelease()
 	rel.package = game1
 	rel.title = "v1.0.0"
 	rel.url = "https://github.com/rubenwardy/capturetheflag/archive/master.zip"
 	rel.approved = True
-	db.session.add(rel)
+	session.add(rel)
 
 
 	mod = Package()
@@ -301,21 +322,22 @@ Uses the CTF PvP Engine.
 	mod.name = "pixelbox"
 	mod.title = "PixelBOX Reloaded"
 	mod.license = licenses["CC0"]
+	mod.media_license = licenses["MIT"]
 	mod.type = PackageType.TXP
-	mod.author = ruben
+	mod.author = admin_user
 	mod.forums = 14132
-	mod.shortDesc = "This is an update of the original PixelBOX texture pack by the brillant artist Gambit"
+	mod.short_desc = "This is an update of the original PixelBOX texture pack by the brillant artist Gambit"
 	mod.desc = "This is the long desc"
-	db.session.add(mod)
+	session.add(mod)
 
 	rel = PackageRelease()
 	rel.package = mod
 	rel.title = "v1.0.0"
 	rel.url = "http://mamadou3.free.fr/Minetest/PixelBOX.zip"
 	rel.approved = True
-	db.session.add(rel)
+	session.add(rel)
 
-	db.session.commit()
+	session.commit()
 
 	metas = {}
 	for package in Package.query.filter_by(type=PackageType.MOD).all():
@@ -324,52 +346,9 @@ Uses the CTF PvP Engine.
 			meta = metas[package.name]
 		except KeyError:
 			meta = MetaPackage(package.name)
-			db.session.add(meta)
+			session.add(meta)
 			metas[package.name] = meta
 		package.provides.append(meta)
 
 	dep = Dependency(food_sweet, meta=metas["food"])
-	db.session.add(dep)
+	session.add(dep)
-
-
-
-delete_db = len(sys.argv) >= 2 and sys.argv[1].strip() == "-d"
-if delete_db and os.path.isfile("db.sqlite"):
-	os.remove("db.sqlite")
-
-print("Creating database tables...")
-db.create_all()
-print("Filling database...")
-
-ruben = User("rubenwardy")
-ruben.github_username = "rubenwardy"
-ruben.forums_username = "rubenwardy"
-ruben.rank = UserRank.ADMIN
-db.session.add(ruben)
-
-tags = {}
-for tag in ["Inventory", "Mapgen", "Building", \
-		"Mobs and NPCs", "Tools", "Player effects", \
-		"Environment", "Transport", "Maintenance", "Plants and farming", \
-		"PvP", "PvE", "Survival", "Creative", "Puzzle", "Multiplayer", "Singleplayer"]:
-	row = Tag(tag)
-	tags[row.name] = row
-	db.session.add(row)
-
-licenses = {}
-for license in ["GPLv2.1", "GPLv3", "LGPLv2.1", "LGPLv3", "AGPLv2.1", "AGPLv3",
-				"Apache", "BSD 3-Clause", "BSD 2-Clause", "CC0", "CC-BY-SA",
-				"CC-BY", "MIT", "ZLib"]:
-	row = License(license)
-	licenses[row.name] = row
-	db.session.add(row)
-
-for license in ["CC-BY-NC-SA"]:
-	row = License(license, False)
-	licenses[row.name] = row
-	db.session.add(row)
-
-if test_data:
-	defineDummyData(licenses, tags, ruben)
-
-db.session.commit()

app/flatpages/help.md

@@ -2,4 +2,6 @@ title: Help
 
 * [Package Tags](package_tags)
 * [Ranks and Permissions](ranks_permissions)
+* [Content Ratings and Flags](content_flags)
 * [Reporting Content](reporting)
+* [API](api)

app/flatpages/help/api.md

@@ -0,0 +1,51 @@
+title: API
+
+## Authentication
+
+Not all endpoints require authentication.
+Authentication is done using Bearer tokens:
+
+	Authorization: Bearer YOURTOKEN
+
+You can use the `/api/whoami` to check authentication.
+
+## Endpoints
+
+### Misc
+
+* GET `/api/whoami/` - Json dictionary with the following keys:
+	* `is_authenticated` - True on successful API authentication
+	* `username` - Username of the user authenticated as, null otherwise.
+	* 403 will be thrown on unsupported authentication type, invalid access token, or other errors.
+
+### Packages
+
+* GET `/api/packages/` - See [Package Queries](#package-queries)
+* GET `/api/packages/<username>/<name>/`
+
+### Topics
+
+* GET `/api/topics/` - Supports [Package Queries](#package-queries), and the following two options:
+    * `show_added` - Show topics which exist as packages, default true.
+	* `show_discarded` - Show topics which have been marked as outdated, default false.
+
+### Minetest
+
+* GET `/api/minetest_versions/`
+
+
+## Package Queries
+
+Example:
+
+	/api/packages/?type=mod&type=game&q=mobs+fun&hide=nonfree&hide=gore
+
+Supported query parameters:
+
+* `type` - Package types (`mod`, `game`, `txp`).
+* `q` - Query string
+* `random` - When present, enable random ordering and ignore `sort`.
+* `hide` - Hide content based on [Content Flags](content_flags).
+* `sort` - Sort by (`name`, `views`, `date`, `score`).
+* `order` - Sort ascending (`Asc`) or descending (`desc`).
+* `protocol_version` - Only show packages supported by this Minetest protocol version.

app/flatpages/help/content_flags.md

@@ -0,0 +1,26 @@
+title: Content Flags
+
+Content flags allow you to hide content based on your preferences.
+The filtering is done server-side, which means that you don't need to update
+your client to use new flags.
+
+## Flags
+
+* `nonfree` - can be used to hide packages which do not qualify as
+	'free software', as defined by the Free Software Foundation.
+* A content rating, given below.
+
+
+## Ratings
+
+Content ratings aren't currently supported by ContentDB.
+Instead, mature content isn't allowed at all for now.
+
+In the future, more mature content will be allowed but labelled with
+content ratings which may contain the following:
+
+* android_default - meta-rating which includes gore and drugs.
+* desktop_default - meta-rating which won't include anything for now.
+* gore - more than just blood
+* drugs
+* swearing

app/flatpages/help/ranks_permissions.md

@@ -34,6 +34,8 @@ title: Ranks and Permissions
 			<th>N</th>
 			<th>Y</th>
 			<th>N</th>
+			<th>Y</th>
+			<th>N</th>
 		</tr>
 	</thead>
 	<tbody>
@@ -58,7 +60,7 @@ title: Ranks and Permissions
 			<th></th>
 			<th></th> <!-- member -->
 			<th></th>
-			<th>✓</th> <!-- trusted member -->
+			<th></th> <!-- trusted member -->
 			<th></th>
 			<th>✓</th> <!-- editor -->
 			<th>✓</th>
@@ -101,7 +103,7 @@ title: Ranks and Permissions
 			<td>Approve Screenshot</td>
 			<th></th> <!-- new -->
 			<th></th>
-			<th>✓</th> <!-- member -->
+			<th></th> <!-- member -->
 			<th></th>
 			<th>✓</th> <!-- trusted member -->
 			<th></th>
@@ -187,6 +189,21 @@ title: Ranks and Permissions
 			<th>✓</th> <!-- admin -->
 			<th>✓</th>
 		</tr>
+		<tr>
+			<td>See Private Thread</td>
+			<th>✓</th> <!-- new -->
+			<th></th>
+			<th>✓</th> <!-- member -->
+			<th></th>
+			<th>✓</th> <!-- trusted member -->
+			<th></th>
+			<th>✓</th> <!-- editor -->
+			<th>✓</th>
+			<th>✓</th> <!-- moderator -->
+			<th>✓</th>
+			<th>✓</th> <!-- admin -->
+			<th>✓</th>
+		</tr>
 		<tr>
 			<td>Set Email</td>
 			<th>✓</th> <!-- new -->
@@ -202,6 +219,21 @@ title: Ranks and Permissions
 			<th>✓</th> <!-- admin -->
 			<th>✓</th>
 		</tr>
+		<tr>
+			<td>Create Token</td>
+			<th></th> <!-- new -->
+			<th></th>
+			<th>✓</th> <!-- member -->
+			<th></th>
+			<th>✓</th> <!-- trusted member -->
+			<th></th>
+			<th>✓</th> <!-- editor -->
+			<th></th>
+			<th>✓</th> <!-- moderator -->
+			<th>✓<sup>2</sup></th>
+			<th>✓</th> <!-- admin -->
+			<th>✓</th>
+		</tr>
 		<tr>
 			<td>Set Rank</td>
 			<th></th> <!-- new -->

app/flatpages/help/reporting.md

@@ -5,4 +5,4 @@ laws.
 
 We take copyright violation and other offenses very seriously.
 
-<a href="https://rubenwardy.com/contact/" class="button btn_green">Contact</a>
+<a href="https://rubenwardy.com/contact/" class="btn btn-success">Contact</a>

app/flatpages/help/wtfpl.md

@@ -0,0 +1,42 @@
+title: WTFPL is a terrible license
+no_h1: true
+
+<div id="warning" class="alert alert-warning">
+	<span class="icon_message"></span>
+
+	Please reconsider the choice of WTFPL as a license.
+
+	<script src="/static/jquery.min.js"></script>
+	<script>
+		// @author rubenwardy
+		// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt GPL-v3-or-Later
+
+		var params = new URLSearchParams(location.search);
+		var r      = params.get("r");
+		if (r)
+			document.write("<a class='alert_right button' href='" + r + "'>Okay</a>");
+		else
+			$("#warning").hide();
+	</script>
+</div>
+
+# WTFPL is a terrible license
+
+The use of WTFPL as a license is discouraged for multiple reasons.
+
+* **No Warranty disclaimer:** This could open you up to being sued.<sup>[1]</sup>
+* **Swearing:** This prevents settings like schools from using your content.
+* **Not OSI Approved:** Same as public domain?
+
+The Open Source Initiative chose not to approve the license as an open-source
+license, saying:<sup>[3]</sup>
+
+> It's no different from dedication to the public domain.
+> Author has submitted license approval request – author is free to make public domain dedication.
+> Although he agrees with the recommendation, Mr. Michlmayr notes that public domain doesn't exist in Europe. Recommend: Reject.
+
+## Sources
+
+1. [WTFPL is harmful to software developers](https://cubicspot.blogspot.com/2017/04/wtfpl-is-harmful-to-software-developers.html)
+2. [FSF](https://www.gnu.org/licenses/license-list.en.html)
+3. [OSI](https://opensource.org/minutes20090304)

app/flatpages/policy_and_guidance.md

@@ -1,14 +1,28 @@
 title: Package Inclusion Policy and Guidance
 
-<div class="box box_grey alert alert-warning">
+<div class="alert alert-warning">
 	<b>Note:</b> This is a draft
 </div>
 
-## 1. General
+## 0. Overview
 
-It is not permitted to submit abusive, obscene, vulgar, slanderous, hateful,
+ContentDB is for the community, and as such listings should be useful to the
-threatening, sexually-orientated or any material that may violate any laws be
+community. To help with this, there are a few rules to improve the quality of
-it of your country, the country where "Content DB” is hosted or International Law.
+the listings and to combat abuse.
+
+* No inappropriate content. <sup>2.1</sup>
+* Content must be playable/useful, but not necessarily finished. <sup>2.2</sup>
+* Don't use the name of another mod unless your mod is a fork or reimplementation. <sup>3</sup>
+* Licenses must allow derivatives, redistribution, and must not discriminate. <sup>4</sup>
+* Don't put promotions or advertisements in package listings, except for
+  donation and personal website links which are permitted in the
+  long description. <sup>5</sup>
+* The ContentDB admin reserves the right to remove packages for any reason,
+  including ones not covered by this document, and to ban users who abuse
+  this service. <sup>1</sup>
+
+
+## 1. General
 
 The ContentDB admin reserves the right to remove packages for any reason,
 including ones not covered by this document, and to ban users who abuse this service.
@@ -16,20 +30,32 @@ including ones not covered by this document, and to ban users who abuse this ser
 Also see the [help page on tags](/help/package_tags/).
 
 
-## 2. Accepted Content and State of Completion
+## 2. Accepted Content
+
+### 2.1. Acceptable Content
+
+Sexually-orientated content is not permitted.
+
+Mature content, including that relating to drugs, excessive gore, violence, or
+excessive horror, is not currently permitted - but will be in the future.
 
 The submission of malware is strictly prohibited. This includes software which
 does not do as it advertises, for example if it posts telemetry without stating
 clearly that it does in the package meta.
 
-ContentDB should only currently contain playable content, ie: stuff that would
+### 2.2. State of Completion
-be in Mod Releases and Game Releases. Please don't upload any Work In Progress (WIP)
-things. This will probably change in future if/when an "early access" feature is
-added.
 
-Adding non-player facing mods, such as libraries and server tools, is perfectly fine.
+ContentDB should only currently contain playable content - content which is
-ContentDB isn't just for player-facing things, and adding libraries allows them to be
+sufficiently complete to be useful to end users. It's fine to add stuff which
-installed when a mod depends on it.
+is still a work in progress (WIP) as long as it adds sufficient value -
+MineClone 2 is a good example of a WIP package which may break between releases
+but still has value. Note that this doesn't mean that you should add a thing
+you started working on yesterday, it's worth adding all the basic stuff to
+make your package useful.
+
+Adding non-player facing mods, such as libraries and server tools, is perfectly fine
+and encouraged. ContentDB isn't just for player-facing things, and adding
+libraries allows them to be installed when a mod depends on it.
 
 
 ## 3. Technical Names
@@ -48,6 +74,8 @@ package and give the name to the correct one.
 If you submit a package where you don't have the right to the name you will be asked
 to change the name of the package, or your package won't be accepted.
 
+We reserve the right to issue exceptions for this where we feel necessary.
+
 ### 3.2 Mod Forks and Reimplementations
 
 An exception to the above is that mods are allowed to have the same name as a
@@ -66,39 +94,46 @@ Please ensure that you correctly credit any resources (code, assets, or otherwis
 that you have used in your package.
 
 **The use of licenses which do not allow derivatives or redistribution is not
-permitted. This includes CC-ND (No-Derivatives) and lots of closed source licenses.**
+permitted. This includes CC-ND (No-Derivatives) and lots of closed source licenses.
+The use of licenses which discriminate between groups of people or forbid the use
+of the content on servers or singleplayer is also not permitted.**
 
 However, closed sourced licenses are allowed if they allow the above.
 
-If the license you use is not on the list then please choose the correct "Other"
+If the license you use is not on the list then please select "Other", and we'll
-option.
+get around to adding it.
 
 Please note that the definitions of "free" and "non-free" is the same as that
 of the [Free Software Foundation](https://www.gnu.org/philosophy/free-sw.en.html).
 
 ### 4.2 Recommended Licenses
 
+It is highly recommended that you use a free and open source software license.
+FOSS licenses result in a sharing community and will increase the number of potential users your package has.
+Using a closed source license will result in your package being massively penalised in the search results and package lists.
+
 It is recommended that you use a proper license for code with a warranty
 disclaimer, such as the (L)GPL or MIT. You should also use a proper media license
 for media, such as a Creative Commons license.
 
-The use of WTFPL is discouraged as it doesn't contain a valid warranty disclaimer,
+The use of WTFPL is discouraged as it doesn't contain a [valid warranty disclaimer](https://cubicspot.blogspot.com/2017/04/wtfpl-is-harmful-to-software-developers.html),
-and also includes swearing which dissuades teachers from using your content.
+and also includes swearing which prevents settings like schools from using your content.
+[Read more](/help/wtfpl/).
 
 Public domain is not a valid license in many countries, please use CC0 or MIT instead.
 
 
 ## 5. Promotions and Advertisements (inc. asking for donations)
 
-Any information other than the long description - including screenshots - must
+You may note place any promotions or advertisements in any meta data including
-not contain any promotions or advertisements. This includes asking for donations,
+screensthos. This includes asking for donations, promoting online shops,
-promoting online shops, or linking to personal websites and social media.
+or linking to personal websites and social media. Please instead use the
+fields provided on your user profile page to place links to websites and
+donation pages.
 
 ContentDB is for the community. We may remove any promotions if we feel that
 they're inappropriate.
 
-Paid promotions are not allowed at all, anywhere.
-
 
 ## 6. Reporting Violations
 

app/maillogger.py

@@ -0,0 +1,109 @@
+import logging
+from enum import Enum
+from app.tasks.emails import sendEmailRaw
+
+def _has_newline(line):
+	"""Used by has_bad_header to check for \\r or \\n"""
+	if line and ("\r" in line or "\n" in line):
+		return True
+	return False
+
+def _is_bad_subject(subject):
+	"""Copied from: flask_mail.py class Message def has_bad_headers"""
+	if _has_newline(subject):
+		for linenum, line in enumerate(subject.split("\r\n")):
+			if not line:
+				return True
+			if linenum > 0 and line[0] not in "\t ":
+				return True
+			if _has_newline(line):
+				return True
+			if len(line.strip()) == 0:
+				return True
+	return False
+
+
+class FlaskMailSubjectFormatter(logging.Formatter):
+	def format(self, record):
+		record.message = record.getMessage()
+		if self.usesTime():
+			record.asctime = self.formatTime(record, self.datefmt)
+		s = self.formatMessage(record)
+		return s
+
+class FlaskMailTextFormatter(logging.Formatter):
+	pass
+
+# TODO: hier nog niet tevreden over (vooral logger.error(..., exc_info, stack_info))
+class FlaskMailHTMLFormatter(logging.Formatter):
+	pre_template = "<h1>%s</h1><pre>%s</pre>"
+	def formatException(self, exc_info):
+		formatted_exception = logging.Handler.formatException(self, exc_info)
+		return FlaskMailHTMLFormatter.pre_template % ("Exception information", formatted_exception)
+	def formatStack(self, stack_info):
+		return FlaskMailHTMLFormatter.pre_template % ("<h1>Stack information</h1><pre>%s</pre>", stack_info)
+
+
+# see: https://github.com/python/cpython/blob/3.6/Lib/logging/__init__.py (class Handler)
+
+class FlaskMailHandler(logging.Handler):
+	def __init__(self, mailer, subject_template, level=logging.NOTSET):
+		logging.Handler.__init__(self, level)
+		self.mailer = mailer
+		self.send_to = mailer.app.config["MAIL_UTILS_ERROR_SEND_TO"]
+		self.subject_template = subject_template
+		self.html_formatter = None
+
+	def setFormatter(self, text_fmt, html_fmt=None):
+		"""
+		Set the formatters for this handler. Provide at least one formatter.
+		When no text_fmt is provided, no text-part is created for the email body.
+		"""
+		assert (text_fmt, html_fmt) != (None, None), "At least one formatter should be provided"
+		if type(text_fmt)==str:
+			text_fmt = FlaskMailTextFormatter(text_fmt)
+		self.formatter = text_fmt
+		if type(html_fmt)==str:
+			html_fmt = FlaskMailHTMLFormatter(html_fmt)
+		self.html_formatter = html_fmt
+
+	def getSubject(self, record):
+		fmt = FlaskMailSubjectFormatter(self.subject_template)
+		subject = fmt.format(record)
+		#Since templates can cause header problems, and we rather have a incomplete email then an error, we fix this
+		if _is_bad_subject(subject):
+			subject="FlaskMailHandler log-entry from %s [original subject is replaced, because it would result in a bad header]" % self.mailer.app.name
+		return subject
+
+	def emit(self, record):
+		text = self.format(record)				if self.formatter	  else None
+		html = self.html_formatter.format(record) if self.html_formatter else None
+		sendEmailRaw.delay(self.send_to, self.getSubject(record), text, html)
+
+
+def register_mail_error_handler(app, mailer):
+	subject_template = "ContentDB crashed (%(module)s > %(funcName)s)"
+	text_template = """
+Message type: %(levelname)s
+Location:	 %(pathname)s:%(lineno)d
+Module:	   %(module)s
+Function:	 %(funcName)s
+Time:		 %(asctime)s
+Message:
+%(message)s"""
+	html_template = """
+<style>th { text-align: right}</style><table>
+<tr><th>Message type:</th><td>%(levelname)s</td></tr>
+<tr>	<th>Location:</th><td>%(pathname)s:%(lineno)d</td></tr>
+<tr>	  <th>Module:</th><td>%(module)s</td></tr>
+<tr>	<th>Function:</th><td>%(funcName)s</td></tr>
+<tr>		<th>Time:</th><td>%(asctime)s</td></tr>
+</table>
+<h2>Message</h2>
+<pre>%(message)s</pre>"""
+
+	import logging
+	mail_handler = FlaskMailHandler(mailer, subject_template)
+	mail_handler.setLevel(logging.ERROR)
+	mail_handler.setFormatter(text_template, html_template)
+	app.logger.addHandler(mail_handler)

app/models.py

@@ -15,19 +15,29 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-from flask import Flask, url_for
+import enum, datetime
-from flask_sqlalchemy import SQLAlchemy
+
-from flask_migrate import Migrate
+from app import app, gravatar
 from urllib.parse import urlparse
-from app import app
+
-from datetime import datetime
+from flask import Flask, url_for
-from sqlalchemy.orm import validates
+from flask_sqlalchemy import SQLAlchemy, BaseQuery
-from flask_user import login_required, UserManager, UserMixin, SQLAlchemyAdapter
+from flask_migrate import Migrate
-import enum
+from flask_user import login_required, UserManager, UserMixin
+from sqlalchemy import func, CheckConstraint
+from sqlalchemy_searchable import SearchQueryMixin
+from sqlalchemy_utils.types import TSVectorType
+from sqlalchemy_searchable import make_searchable
+
 
 # Initialise database
 db = SQLAlchemy(app)
 migrate = Migrate(app, db)
+make_searchable(db.metadata)
+
+
+class ArticleQuery(BaseQuery, SearchQueryMixin):
+	pass
 
 
 class UserRank(enum.Enum):
@@ -66,7 +76,9 @@ class Permission(enum.Enum):
 	APPROVE_CHANGES    = "APPROVE_CHANGES"
 	DELETE_PACKAGE     = "DELETE_PACKAGE"
 	CHANGE_AUTHOR      = "CHANGE_AUTHOR"
+	CHANGE_NAME        = "CHANGE_NAME"
 	MAKE_RELEASE       = "MAKE_RELEASE"
+	DELETE_RELEASE     = "DELETE_RELEASE"
 	ADD_SCREENSHOTS    = "ADD_SCREENSHOTS"
 	APPROVE_SCREENSHOT = "APPROVE_SCREENSHOT"
 	APPROVE_RELEASE    = "APPROVE_RELEASE"
@@ -77,6 +89,10 @@ class Permission(enum.Enum):
 	CHANGE_EMAIL       = "CHANGE_EMAIL"
 	EDIT_EDITREQUEST   = "EDIT_EDITREQUEST"
 	SEE_THREAD         = "SEE_THREAD"
+	CREATE_THREAD      = "CREATE_THREAD"
+	UNAPPROVE_PACKAGE  = "UNAPPROVE_PACKAGE"
+	TOPIC_DISCARD      = "TOPIC_DISCARD"
+	CREATE_TOKEN       = "CREATE_TOKEN"
 
 	# Only return true if the permission is valid for *all* contexts
 	# See Package.checkPerm for package-specific contexts
@@ -85,34 +101,40 @@ class Permission(enum.Enum):
 			return False
 
 		if self == Permission.APPROVE_NEW or \
-				self == Permission.APPROVE_CHANGES or \
+				self == Permission.APPROVE_CHANGES    or \
-				self == Permission.APPROVE_RELEASE or \
+				self == Permission.APPROVE_RELEASE    or \
-				self == Permission.APPROVE_SCREENSHOT:
+				self == Permission.APPROVE_SCREENSHOT or \
+				self == Permission.SEE_THREAD:
 			return user.rank.atLeast(UserRank.EDITOR)
 		else:
 			raise Exception("Non-global permission checked globally. Use Package.checkPerm or User.checkPerm instead.")
 
 class User(db.Model, UserMixin):
-	id = db.Column(db.Integer, primary_key=True)
+	id           = db.Column(db.Integer, primary_key=True)
 
 	# User authentication information
-	username = db.Column(db.String(50, collation="NOCASE"), nullable=False, unique=True, index=True)
+	username     = db.Column(db.String(50, collation="NOCASE"), nullable=False, unique=True, index=True)
-	password = db.Column(db.String(255), nullable=True)
+	password     = db.Column(db.String(255), nullable=False, server_default="")
 	reset_password_token = db.Column(db.String(100), nullable=False, server_default="")
 
-	rank = db.Column(db.Enum(UserRank))
+	rank         = db.Column(db.Enum(UserRank))
 
 	# Account linking
 	github_username = db.Column(db.String(50, collation="NOCASE"), nullable=True, unique=True)
 	forums_username = db.Column(db.String(50, collation="NOCASE"), nullable=True, unique=True)
 
 	# User email information
-	email = db.Column(db.String(255), nullable=True, unique=True)
+	email         = db.Column(db.String(255), nullable=True, unique=True)
-	confirmed_at = db.Column(db.DateTime())
+	email_confirmed_at  = db.Column(db.DateTime())
 
 	# User information
-	active = db.Column("is_active", db.Boolean, nullable=False, server_default="0")
+	profile_pic   = db.Column(db.String(255), nullable=True, server_default=None)
-	display_name = db.Column(db.String(100), nullable=False, server_default="")
+	active        = db.Column("is_active", db.Boolean, nullable=False, server_default="0")
+	display_name  = db.Column(db.String(100), nullable=False, server_default="")
+
+	# Links
+	website_url   = db.Column(db.String(255), nullable=True, default=None)
+	donate_url    = db.Column(db.String(255), nullable=True, default=None)
 
 	# Content
 	notifications = db.relationship("Notification", primaryjoin="User.id==Notification.user_id")
@@ -121,19 +143,21 @@ class User(db.Model, UserMixin):
 	packages      = db.relationship("Package", backref="author", lazy="dynamic")
 	requests      = db.relationship("EditRequest", backref="author", lazy="dynamic")
 	threads       = db.relationship("Thread", backref="author", lazy="dynamic")
+	tokens        = db.relationship("APIToken", backref="owner", lazy="dynamic")
 	replies       = db.relationship("ThreadReply", backref="author", lazy="dynamic")
 
-	def __init__(self, username, active=False, email=None, password=None):
+	def __init__(self, username, active=False, email=None, password=""):
-		import datetime
-
 		self.username = username
-		self.confirmed_at = datetime.datetime.now() - datetime.timedelta(days=6000)
+		self.email_confirmed_at = datetime.datetime.now() - datetime.timedelta(days=6000)
 		self.display_name = username
 		self.active = active
 		self.email = email
 		self.password = password
 		self.rank = UserRank.NOT_JOINED
 
+	def hasPassword(self):
+		return self.password != ""
+
 	def canAccessTodoList(self):
 		return Permission.APPROVE_NEW.check(self) or \
 				Permission.APPROVE_RELEASE.check(self) or \
@@ -142,6 +166,12 @@ class User(db.Model, UserMixin):
 	def isClaimed(self):
 		return self.rank.atLeast(UserRank.NEW_MEMBER)
 
+	def getProfilePicURL(self):
+		if self.profile_pic:
+			return self.profile_pic
+		else:
+			return gravatar(self.email or "")
+
 	def checkPerm(self, user, perm):
 		if not user.is_authenticated:
 			return False
@@ -158,9 +188,31 @@ class User(db.Model, UserMixin):
 			return user.rank.atLeast(UserRank.MODERATOR)
 		elif perm == Permission.CHANGE_EMAIL:
 			return user == self or (user.rank.atLeast(UserRank.MODERATOR) and user.rank.atLeast(self.rank))
+		elif perm == Permission.CREATE_TOKEN:
+			if user == self:
+				return user.rank.atLeast(UserRank.MEMBER)
+			else:
+				return user.rank.atLeast(UserRank.MODERATOR) and user.rank.atLeast(self.rank)
 		else:
 			raise Exception("Permission {} is not related to users".format(perm.name))
 
+	def canCommentRL(self):
+		hour_ago = datetime.datetime.utcnow() - datetime.timedelta(hours=1)
+		return ThreadReply.query.filter_by(author=self) \
+			.filter(ThreadReply.created_at > hour_ago).count() < 4
+
+	def canOpenThreadRL(self):
+		hour_ago = datetime.datetime.utcnow() - datetime.timedelta(hours=1)
+		return Thread.query.filter_by(author=self) \
+			.filter(Thread.created_at > hour_ago).count() < 2
+
+	def __eq__(self, other):
+		if not self.is_authenticated or not other.is_authenticated:
+			return False
+
+		assert self.id > 0
+		return self.id == other.id
+
 class UserEmailVerification(db.Model):
 	id      = db.Column(db.Integer, primary_key=True)
 	user_id = db.Column(db.Integer, db.ForeignKey("user.id"))
@@ -179,6 +231,9 @@ class Notification(db.Model):
 	url       = db.Column(db.String(200), nullable=True)
 
 	def __init__(self, us, cau, titl, ur):
+		if len(titl) > 100:
+			title = title[:99] + "…"
+
 		self.user   = us
 		self.causer = cau
 		self.title  = titl
@@ -209,6 +264,13 @@ class PackageType(enum.Enum):
 	def __str__(self):
 		return self.name
 
+	@classmethod
+	def get(cls, name):
+		try:
+			return PackageType[name.upper()]
+		except KeyError:
+			return None
+
 	@classmethod
 	def choices(cls):
 		return [(choice, choice.value) for choice in cls]
@@ -221,7 +283,7 @@ class PackageType(enum.Enum):
 class PackagePropertyKey(enum.Enum):
 	name          = "Name"
 	title         = "Title"
-	shortDesc     = "Short Description"
+	short_desc     = "Short Description"
 	desc          = "Description"
 	type          = "Type"
 	license       = "License"
@@ -258,7 +320,7 @@ class Dependency(db.Model):
 	package         = db.relationship("Package", foreign_keys=[package_id])
 	meta_package_id = db.Column(db.Integer, db.ForeignKey("meta_package.id"), nullable=True)
 	optional        = db.Column(db.Boolean, nullable=False, default=False)
-	__table_args__  = (db.UniqueConstraint('depender_id', 'package_id', 'meta_package_id', name='_dependency_uc'), )
+	__table_args__  = (db.UniqueConstraint("depender_id", "package_id", "meta_package_id", name="_dependency_uc"), )
 
 	def __init__(self, depender=None, package=None, meta=None):
 		if depender is None:
@@ -318,18 +380,24 @@ class Dependency(db.Model):
 		return retval
 
 
-
 class Package(db.Model):
+	query_class  = ArticleQuery
+
 	id           = db.Column(db.Integer, primary_key=True)
 
 	# Basic details
 	author_id    = db.Column(db.Integer, db.ForeignKey("user.id"))
-	name         = db.Column(db.String(100), nullable=False)
+	name         = db.Column(db.Unicode(100), nullable=False)
-	title        = db.Column(db.String(100), nullable=False)
+	title        = db.Column(db.Unicode(100), nullable=False)
-	shortDesc    = db.Column(db.String(200), nullable=False)
+	short_desc   = db.Column(db.Unicode(200), nullable=False)
-	desc         = db.Column(db.Text, nullable=True)
+	desc         = db.Column(db.UnicodeText, nullable=True)
 	type         = db.Column(db.Enum(PackageType))
-	created_at   = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
+	created_at   = db.Column(db.DateTime, nullable=False, default=datetime.datetime.utcnow)
+
+	name_valid = db.CheckConstraint("name ~* '^[a-z0-9_]+$'")
+
+	search_vector = db.Column(TSVectorType("name", "title", "short_desc", "desc", \
+			weights={ "name": "A", "title": "B", "short_desc": "C", "desc": "D" }))
 
 	license_id   = db.Column(db.Integer, db.ForeignKey("license.id"), nullable=False, default=1)
 	license      = db.relationship("License", foreign_keys=[license_id])
@@ -339,6 +407,8 @@ class Package(db.Model):
 	approved     = db.Column(db.Boolean, nullable=False, default=False)
 	soft_deleted = db.Column(db.Boolean, nullable=False, default=False)
 
+	score        = db.Column(db.Float, nullable=False, default=0)
+
 	review_thread_id = db.Column(db.Integer, db.ForeignKey("thread.id"), nullable=True, default=None)
 	review_thread    = db.relationship("Thread", foreign_keys=[review_thread_id])
 
@@ -349,7 +419,7 @@ class Package(db.Model):
 	forums       = db.Column(db.Integer,     nullable=True)
 
 	provides = db.relationship("MetaPackage", secondary=provides, lazy="subquery",
-			backref=db.backref("packages", lazy="dynamic"))
+			backref=db.backref("packages", lazy="dynamic", order_by=db.desc("score")))
 
 	dependencies = db.relationship("Dependency", backref="depender", lazy="dynamic", foreign_keys=[Dependency.depender_id])
 
@@ -360,7 +430,7 @@ class Package(db.Model):
 			lazy="dynamic", order_by=db.desc("package_release_releaseDate"))
 
 	screenshots = db.relationship("PackageScreenshot", backref="package",
-			lazy="dynamic")
+			lazy="dynamic", order_by=db.asc("package_screenshot_id"))
 
 	requests = db.relationship("EditRequest", backref="package",
 			lazy="dynamic")
@@ -376,35 +446,63 @@ class Package(db.Model):
 		for e in PackagePropertyKey:
 			setattr(self, e.name, getattr(package, e.name))
 
-	def getAsDictionaryShort(self, base_url):
+	def getIsFOSS(self):
-		tnurl = self.getThumbnailURL()
+		return self.license.is_foss and self.media_license.is_foss
+
+	def getState(self):
+		if self.approved:
+			return "approved"
+		elif self.review_thread_id:
+			return "thread"
+		elif (self.type == PackageType.GAME or \
+					self.type == PackageType.TXP) and \
+				self.screenshots.count() == 0:
+			return "wip"
+		elif not self.getDownloadRelease():
+			return "wip"
+		elif "Other" in self.license.name or "Other" in self.media_license.name:
+			return "license"
+		else:
+			return "ready"
+
+	def getAsDictionaryKey(self):
+		return {
+			"name": self.name,
+			"author": self.author.display_name,
+			"type": self.type.toName(),
+		}
+
+	def getAsDictionaryShort(self, base_url, version=None, protonum=None):
+		tnurl = self.getThumbnailURL(1)
+		release = self.getDownloadRelease(version=version, protonum=protonum)
 		return {
 			"name": self.name,
 			"title": self.title,
 			"author": self.author.display_name,
-			"shortDesc": self.shortDesc,
+			"short_description": self.short_desc,
 			"type": self.type.toName(),
-			"release": self.getDownloadRelease().id if self.getDownloadRelease() is not None else None,
+			"release": release and release.id,
 			"thumbnail": (base_url + tnurl) if tnurl is not None else None
 		}
 
-	def getAsDictionary(self, base_url):
+	def getAsDictionary(self, base_url, version=None, protonum=None):
-		tnurl = self.getThumbnailURL()
+		tnurl = self.getThumbnailURL(1)
+		release = self.getDownloadRelease(version=version, protonum=protonum)
 		return {
 			"author": self.author.display_name,
 			"name": self.name,
 			"title": self.title,
-			"shortDesc": self.shortDesc,
+			"short_description": self.short_desc,
 			"desc": self.desc,
 			"type": self.type.toName(),
-			"createdAt": self.created_at,
+			"created_at": self.created_at,
 
 			"license": self.license.name,
-			"mediaLicense": self.media_license.name,
+			"media_license": self.media_license.name,
 
 			"repo": self.repo,
 			"website": self.website,
-			"issueTracker": self.issueTracker,
+			"issue_tracker": self.issueTracker,
 			"forums": self.forums,
 
 			"provides": [x.name for x in self.provides],
@@ -412,56 +510,78 @@ class Package(db.Model):
 			"screenshots": [base_url + ss.url for ss in self.screenshots],
 
 			"url": base_url + self.getDownloadURL(),
-			"release": self.getDownloadRelease().id if self.getDownloadRelease() is not None else None
+			"release": release and release.id,
+
+			"score": round(self.score * 10) / 10
 		}
 
-	def getThumbnailURL(self):
+	def getThumbnailURL(self, level=2):
-		screenshot = self.screenshots.filter_by(approved=True).first()
+		screenshot = self.screenshots.filter_by(approved=True).order_by(db.asc(PackageScreenshot.id)).first()
-		return screenshot.getThumbnailURL() if screenshot is not None else None
+		return screenshot.getThumbnailURL(level) if screenshot is not None else None
 
 	def getMainScreenshotURL(self):
-		screenshot = self.screenshots.filter_by(approved=True).first()
+		screenshot = self.screenshots.filter_by(approved=True).order_by(db.asc(PackageScreenshot.id)).first()
 		return screenshot.url if screenshot is not None else None
 
 	def getDetailsURL(self):
-		return url_for("package_page",
+		return url_for("packages.view",
 				author=self.author.username, name=self.name)
 
 	def getEditURL(self):
-		return url_for("create_edit_package_page",
+		return url_for("packages.create_edit",
 				author=self.author.username, name=self.name)
 
 	def getApproveURL(self):
-		return url_for("approve_package_page",
+		return url_for("packages.approve",
 				author=self.author.username, name=self.name)
 
-	def getDeleteURL(self):
+	def getRemoveURL(self):
-		return url_for("delete_package_page",
+		return url_for("packages.remove",
 				author=self.author.username, name=self.name)
 
 	def getNewScreenshotURL(self):
-		return url_for("create_screenshot_page",
+		return url_for("packages.create_screenshot",
 				author=self.author.username, name=self.name)
 
 	def getCreateReleaseURL(self):
-		return url_for("create_release_page",
+		return url_for("packages.create_release",
 				author=self.author.username, name=self.name)
 
 	def getCreateEditRequestURL(self):
 		return url_for("create_edit_editrequest_page",
 				author=self.author.username, name=self.name)
 
+	def getBulkReleaseURL(self):
+		return url_for("packages.bulk_change_release",
+			author=self.author.username, name=self.name)
+
 	def getDownloadURL(self):
-		return url_for("package_download_page",
+		return url_for("packages.download",
 				author=self.author.username, name=self.name)
 
-	def getDownloadRelease(self):
+	def getDownloadRelease(self, version=None, protonum=None):
+		if version is None and protonum is not None:
+			version = MinetestRelease.query.filter(MinetestRelease.protocol >= int(protonum)).first()
+			if version is not None:
+				version = version.id
+			else:
+				version = 10000000
+
+
 		for rel in self.releases:
-			if rel.approved:
+			if rel.approved and (version is None or
+					((rel.min_rel is None or rel.min_rel_id <= version) and \
+					(rel.max_rel is None or rel.max_rel_id >= version))):
 				return rel
 
 		return None
 
+	def getDownloadCount(self):
+		counter = 0
+		for release in self.releases:
+			counter += release.downloads
+		return counter
+
 	def checkPerm(self, user, perm):
 		if not user.is_authenticated:
 			return False
@@ -473,6 +593,9 @@ class Package(db.Model):
 
 		isOwner = user == self.author
 
+		if perm == Permission.CREATE_THREAD:
+			return user.rank.atLeast(UserRank.MEMBER)
+
 		# Members can edit their own packages, and editors can edit any packages
 		if perm == Permission.MAKE_RELEASE or perm == Permission.ADD_SCREENSHOTS:
 			return isOwner or user.rank.atLeast(UserRank.EDITOR)
@@ -483,21 +606,47 @@ class Package(db.Model):
 			else:
 				return user.rank.atLeast(UserRank.EDITOR)
 
-		# Editors can change authors
+		# Anyone can change the package name when not approved, but only editors when approved
-		elif perm == Permission.CHANGE_AUTHOR:
+		elif perm == Permission.CHANGE_NAME:
+			return not self.approved or user.rank.atLeast(UserRank.EDITOR)
+
+		# Editors can change authors and approve new packages
+		elif perm == Permission.APPROVE_NEW or perm == Permission.CHANGE_AUTHOR:
 			return user.rank.atLeast(UserRank.EDITOR)
 
-		elif perm == Permission.APPROVE_NEW or perm == Permission.APPROVE_RELEASE \
+		elif perm == Permission.APPROVE_RELEASE or perm == Permission.APPROVE_SCREENSHOT:
-				or perm == Permission.APPROVE_SCREENSHOT:
 			return user.rank.atLeast(UserRank.TRUSTED_MEMBER if isOwner else UserRank.EDITOR)
 
 		# Moderators can delete packages
-		elif perm == Permission.DELETE_PACKAGE or perm == Permission.CHANGE_RELEASE_URL:
+		elif perm == Permission.DELETE_PACKAGE or perm == Permission.UNAPPROVE_PACKAGE \
+				or perm == Permission.CHANGE_RELEASE_URL:
 			return user.rank.atLeast(UserRank.MODERATOR)
 
 		else:
 			raise Exception("Permission {} is not related to packages".format(perm.name))
 
+	def setStartScore(self):
+		downloads = db.session.query(func.sum(PackageRelease.downloads)). \
+				filter(PackageRelease.package_id == self.id).scalar() or 0
+
+		forum_score = 0
+		forum_bonus = 0
+		topic = self.forums and ForumTopic.query.get(self.forums)
+		if topic:
+			months = (datetime.datetime.now() - topic.created_at).days / 30
+			years  = months / 12
+			forum_score = topic.views / max(years, 0.0416) + 80*min(max(months, 0.5), 6)
+			forum_bonus = topic.views + topic.posts
+
+		self.score = max(downloads, forum_score * 0.6) + forum_bonus
+
+		if self.getMainScreenshotURL() is None:
+			self.score *= 0.8
+
+		if not self.license.is_foss or not self.media_license.is_foss:
+			self.score *= 0.1
+
+
 class MetaPackage(db.Model):
 	id           = db.Column(db.Integer, primary_key=True)
 	name         = db.Column(db.String(100), unique=True, nullable=False)
@@ -562,25 +711,110 @@ class Tag(db.Model):
 		regex = re.compile("[^a-z_]")
 		self.name = regex.sub("", self.title.lower().replace(" ", "_"))
 
+
+class MinetestRelease(db.Model):
+	id       = db.Column(db.Integer, primary_key=True)
+	name     = db.Column(db.String(100), unique=True, nullable=False)
+	protocol = db.Column(db.Integer, nullable=False, default=0)
+
+	def __init__(self, name=None, protocol=0):
+		self.name = name
+		self.protocol = protocol
+
+	def getActual(self):
+		return None if self.name == "None" else self
+
+
 class PackageRelease(db.Model):
 	id           = db.Column(db.Integer, primary_key=True)
 
 	package_id   = db.Column(db.Integer, db.ForeignKey("package.id"))
 	title        = db.Column(db.String(100), nullable=False)
-	releaseDate  = db.Column(db.DateTime,        nullable=False)
+	releaseDate  = db.Column(db.DateTime,    nullable=False)
 	url          = db.Column(db.String(200), nullable=False)
 	approved     = db.Column(db.Boolean, nullable=False, default=False)
 	task_id      = db.Column(db.String(37), nullable=True)
+	commit_hash  = db.Column(db.String(41), nullable=True, default=None)
+	downloads    = db.Column(db.Integer, nullable=False, default=0)
 
+	min_rel_id = db.Column(db.Integer, db.ForeignKey("minetest_release.id"), nullable=True, server_default=None)
+	min_rel    = db.relationship("MinetestRelease", foreign_keys=[min_rel_id])
+
+	max_rel_id = db.Column(db.Integer, db.ForeignKey("minetest_release.id"), nullable=True, server_default=None)
+	max_rel    = db.relationship("MinetestRelease", foreign_keys=[max_rel_id])
+
+	# If the release is approved, then the task_id must be null and the url must be present
+	CK_approval_valid = db.CheckConstraint("not approved OR (task_id IS NULL AND (url = '') IS NOT FALSE)")
 
 	def getEditURL(self):
-		return url_for("edit_release_page",
+		return url_for("packages.edit_release",
 				author=self.package.author.username,
 				name=self.package.name,
 				id=self.id)
 
+	def getDeleteURL(self):
+		return url_for("packages.delete_release",
+				author=self.package.author.username,
+				name=self.package.name,
+				id=self.id)
+
+	def getDownloadURL(self):
+		return url_for("packages.download_release",
+				author=self.package.author.username,
+				name=self.package.name,
+				id=self.id)
+
+
 	def __init__(self):
-		self.releaseDate = datetime.now()
+		self.releaseDate = datetime.datetime.now()
+
+	def approve(self, user):
+		if self.package.approved and \
+				not self.package.checkPerm(user, Permission.APPROVE_RELEASE):
+			return False
+
+		assert self.task_id is None and self.url is not None and self.url != ""
+
+		self.approved = True
+		return True
+
+	def checkPerm(self, user, perm):
+		if not user.is_authenticated:
+			return False
+
+		if type(perm) == str:
+			perm = Permission[perm]
+		elif type(perm) != Permission:
+			raise Exception("Unknown permission given to PackageRelease.checkPerm()")
+
+		isOwner = user == self.package.author
+
+		if perm == Permission.DELETE_RELEASE:
+			if user.rank.atLeast(UserRank.ADMIN):
+				return True
+
+			if not (isOwner or user.rank.atLeast(UserRank.EDITOR)):
+				return False
+
+			if not self.package.approved or self.task_id is not None:
+				return True
+
+			count = PackageRelease.query \
+					.filter_by(package_id=self.package_id) \
+					.filter(PackageRelease.id > self.id) \
+					.count()
+
+			return count > 0
+		else:
+			raise Exception("Permission {} is not related to releases".format(perm.name))
+
+
+# class PackageReview(db.Model):
+# 	id         = db.Column(db.Integer, primary_key=True)
+# 	package_id = db.Column(db.Integer, db.ForeignKey("package.id"))
+# 	thread_id  = db.Column(db.Integer, db.ForeignKey("thread.id"), nullable=False)
+# 	recommend  = db.Column(db.Boolean, nullable=False, default=True)
+
 
 class PackageScreenshot(db.Model):
 	id         = db.Column(db.Integer, primary_key=True)
@@ -591,13 +825,25 @@ class PackageScreenshot(db.Model):
 
 
 	def getEditURL(self):
-		return url_for("edit_screenshot_page",
+		return url_for("packages.edit_screenshot",
 				author=self.package.author.username,
 				name=self.package.name,
 				id=self.id)
 
-	def getThumbnailURL(self):
+	def getThumbnailURL(self, level=2):
-		return self.url.replace("/uploads/", "/thumbnails/350x233/")
+		return self.url.replace("/uploads/", ("/thumbnails/{:d}/").format(level))
+
+
+class APIToken(db.Model):
+	id           = db.Column(db.Integer, primary_key=True)
+	access_token = db.Column(db.String(34), unique=True)
+	name         = db.Column(db.String(100), nullable=False)
+	owner_id     = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False)
+	created_at   = db.Column(db.DateTime, nullable=False, default=datetime.datetime.utcnow)
+
+	def canOperateOnPackage(self, package):
+		return packages.count() == 0 or package in packages
+
 
 class EditRequest(db.Model):
 	id           = db.Column(db.Integer, primary_key=True)
@@ -702,13 +948,22 @@ class Thread(db.Model):
 	title      = db.Column(db.String(100), nullable=False)
 	private    = db.Column(db.Boolean, server_default="0")
 
-	created_at = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
+	created_at = db.Column(db.DateTime, nullable=False, default=datetime.datetime.utcnow)
 
 	replies    = db.relationship("ThreadReply", backref="thread", lazy="dynamic")
 
 	watchers   = db.relationship("User", secondary=watchers, lazy="subquery", \
 						backref=db.backref("watching", lazy=True))
 
+
+	def getSubscribeURL(self):
+		return url_for("threads.subscribe",
+				id=self.id)
+
+	def getUnsubscribeURL(self):
+		return url_for("threads.unsubscribe",
+				id=self.id)
+
 	def checkPerm(self, user, perm):
 		if not user.is_authenticated:
 			return not self.private
@@ -731,11 +986,7 @@ class ThreadReply(db.Model):
 	thread_id  = db.Column(db.Integer, db.ForeignKey("thread.id"), nullable=False)
 	comment    = db.Column(db.String(500), nullable=False)
 	author_id  = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False)
-	created_at = db.Column(db.DateTime, nullable=False, default=datetime.utcnow)
+	created_at = db.Column(db.DateTime, nullable=False, default=datetime.datetime.utcnow)
-
-
-
-
 
 
 REPO_BLACKLIST = [".zip", "mediafire.com", "dropbox.com", "weebly.com", \
@@ -743,30 +994,64 @@ REPO_BLACKLIST = [".zip", "mediafire.com", "dropbox.com", "weebly.com", \
 		"digitalaudioconcepts.com", "hg.intevation.org", "www.wtfpl.net", \
 		"imageshack.com", "imgur.com"]
 
-class KrockForumTopic(db.Model):
+class ForumTopic(db.Model):
 	topic_id  = db.Column(db.Integer, primary_key=True, autoincrement=False)
 	author_id = db.Column(db.Integer, db.ForeignKey("user.id"), nullable=False)
 	author    = db.relationship("User")
 
-	ttype     = db.Column(db.Integer, nullable=False)
+	wip       = db.Column(db.Boolean, server_default="0")
+	discarded = db.Column(db.Boolean, server_default="0")
+
+	type      = db.Column(db.Enum(PackageType), nullable=False)
 	title     = db.Column(db.String(200), nullable=False)
 	name      = db.Column(db.String(30), nullable=True)
 	link      = db.Column(db.String(200), nullable=True)
 
-	def getType(self):
+	posts     = db.Column(db.Integer, nullable=False)
-		if self.ttype == 1 or self.ttype == 2:
+	views     = db.Column(db.Integer, nullable=False)
-			return PackageType.MOD
+
-		elif self.ttype == 6:
+	created_at = db.Column(db.DateTime, nullable=False, default=datetime.datetime.utcnow)
-			return PackageType.GAME
 
 	def getRepoURL(self):
+		if self.link is None:
+			return None
+
 		for item in REPO_BLACKLIST:
 			if item in self.link:
 				return None
 
 		return self.link.replace("repo.or.cz/w/", "repo.or.cz/")
 
+	def getAsDictionary(self):
+		return {
+			"author": self.author.username,
+			"name":   self.name,
+			"type":   self.type.toName(),
+			"title":  self.title,
+			"id":     self.topic_id,
+			"link":   self.link,
+			"posts":  self.posts,
+			"views":  self.views,
+			"is_wip": self.wip,
+			"discarded":  self.discarded,
+			"created_at": self.created_at.isoformat(),
+		}
+
+	def checkPerm(self, user, perm):
+		if not user.is_authenticated:
+			return False
+
+		if type(perm) == str:
+			perm = Permission[perm]
+		elif type(perm) != Permission:
+			raise Exception("Unknown permission given to ForumTopic.checkPerm()")
+
+		if perm == Permission.TOPIC_DISCARD:
+			return self.author == user or user.rank.atLeast(UserRank.EDITOR)
+
+		else:
+			raise Exception("Permission {} is not related to topics".format(perm.name))
+
 
 # Setup Flask-User
-db_adapter = SQLAlchemyAdapter(db, User)        # Register the User model
+user_manager = UserManager(app, db, User)
-user_manager = UserManager(db_adapter, app)     # Initialize Flask-User

app/public/static/opensearch.xml

@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
+	<ShortName>ContentDB</ShortName>
+	<LongName>ContentDB</LongName>
+	<InputEncoding>UTF-8</InputEncoding>
+	<Description>Search mods, games, and textures for Minetest.</Description>
+	<Tags>Minetest Mod Game Subgame Search</Tags>
+	<Url type="text/html" method="get" template="https://content.minetest.net/packages?q={searchTerms}"/>
+</OpenSearchDescription>

app/public/static/package_create.js

@@ -11,6 +11,8 @@ $(function() {
 
 	$(".pkg_meta").hide()
 	$(".pkg_wiz_1").show()
+
+	$("#pkg_wiz_1_skip").click(finish)
 	$("#pkg_wiz_1_next").click(function() {
 		const repoURL = $("#repo").val();
 		if (repoURL.trim() != "") {
@@ -18,41 +20,38 @@ $(function() {
 			$(".pkg_wiz_2").show()
 			$(".pkg_repo").hide()
 
-			function setSpecial(id, value) {
+			function setField(id, value) {
-				if (value != "") {
+				if (value && value != "") {
 					var ele = $(id);
 					ele.val(value);
-					ele.trigger("change")
+					ele.trigger("change");
 				}
 			}
 
 			performTask("/tasks/getmeta/new/?url=" + encodeURI(repoURL)).then(function(result) {
-				$("#name").val(result.name)
+				setField("#name", result.name);
-				setSpecial("#provides_str", result.provides)
+				setField("#provides_str", result.provides);
-				$("#title").val(result.title)
+				setField("#title", result.title);
-				$("#repo").val(result.repo || repoURL)
+				setField("#repo", result.repo || repoURL);
-				$("#issueTracker").val(result.issueTracker)
+				setField("#issueTracker", result.issueTracker);
-				$("#desc").val(result.description)
+				setField("#desc", result.description);
-				$("#shortDesc").val(result.short_description)
+				setField("#short_desc", result.short_description);
-				setSpecial("#harddep_str", result.depends)
+				setField("#harddep_str", result.depends);
-				setSpecial("#softdep_str", result.optional_depends)
+				setField("#softdep_str", result.optional_depends);
-				$("#shortDesc").val(result.short_description)
+				setField("#short_desc", result.short_description);
-				if (result.forumId) {
+				setField("#forums", result.forumId);
-					$("#forums").val(result.forumId)
-				}
-
 				if (result.type && result.type.length > 2) {
-					$("#type").val(result.type)
+					$("#type").val(result.type);
 				}
 
-				finish()
+				finish();
 			}).catch(function(e) {
-				alert(e)
+				alert(e);
-				$(".pkg_wiz_1").show()
+				$(".pkg_wiz_1").show();
-				$(".pkg_wiz_2").hide()
+				$(".pkg_wiz_2").hide();
-				$(".pkg_repo").show()
+				$(".pkg_repo").show();
 				// finish()
-			})
+			});
 		} else {
 			finish()
 		}

app/public/static/package_edit.js

@@ -8,4 +8,57 @@ $(function() {
 	})
 	$(".not_mod, .not_game, .not_txp").show()
 	$(".not_" + $("#type").val().toLowerCase()).hide()
+
+	$("#forums").on('paste', function(e) {
+		try {
+			var pasteData = e.originalEvent.clipboardData.getData('text')
+			var url = new URL(pasteData);
+			if (url.hostname == "forum.minetest.net") {
+				$(this).val(url.searchParams.get("t"));
+				e.preventDefault();
+			}
+		} catch (e) {
+			console.log("Not a URL");
+		}
+	});
+
+	let hint = null;
+	function showHint(ele, text) {
+		if (hint) {
+			hint.remove();
+		}
+
+		hint = ele.parent()
+				.append(`<div class="alert alert-warning my-3">${text}</div>`)
+				.find(".alert");
+	}
+
+	let hint_mtmods = `Tip:
+		Don't include <i>Minetest</i>, <i>mod</i>, or <i>modpack</i> anywhere in the short description.
+		It is unnecessary and wastes characters.`;
+
+	let hint_thegame = `Tip:
+		It's obvious that this adds something to Minetest,
+		there's no need to use phrases such as \"adds X to the game\".`
+
+	$("#short_desc").on("change paste keyup", function() {
+		var val = $(this).val().toLowerCase();
+		if (val.indexOf("minetest") >= 0 || val.indexOf("mod") >= 0 ||
+				val.indexOf("modpack") >= 0 || val.indexOf("mod pack") >= 0) {
+			showHint($(this), hint_mtmods);
+		} else if (val.indexOf("the game") >= 0) {
+			showHint($(this), hint_thegame);
+		} else if (hint) {
+			hint.remove();
+			hint = null;
+		}
+	})
+
+	var btn = $("#forums").parent().find("label").append("<a class='ml-3 btn btn-sm btn-primary'>Open</a>");
+	btn.click(function() {
+		var id = $("#forums").val();
+		if (/^\d+$/.test(id)) {
+			window.open("https://forum.minetest.net/viewtopic.php?t=" + id, "_blank");
+		}
+	});
 })

app/public/static/release_minmax.js

@@ -0,0 +1,18 @@
+var min = $("#min_rel");
+var max = $("#max_rel");
+var none = $("#min_rel option:first-child").attr("value");
+var warning = $("#minmax_warning");
+
+function ver_check() {
+	var minv = min.val();
+	var maxv = max.val();
+
+	if (minv != none && maxv != none && minv > maxv) {
+		warning.show();
+	} else {
+		warning.hide();
+	}
+}
+
+min.change(ver_check);
+max.change(ver_check);

app/public/static/tagselector.js

@@ -5,13 +5,25 @@
  * https://petprojects.googlecode.com/svn/trunk/GPL-LICENSE.txt
  */
 (function($) {
+	function hide_error(input) {
+		var err = input.parent().parent().find(".invalid-remaining");
+		err.hide();
+	}
+
+	function show_error(input, msg) {
+		var err = input.parent().parent().find(".invalid-remaining");
+		console.log(err.length);
+		err.text(msg);
+		err.show();
+	}
+
 	$.fn.selectSelector = function(source, name, select) {
 		return this.each(function() {
 				var selector = $(this),
 					input = $('input[type=text]', this);
 
 				selector.click(function() { input.focus(); })
-					.delegate('.tag a', 'click', function() {
+					.delegate('.badge a', 'click', function() {
 						var id = $(this).parent().data("id");
 						for (var i = 0; i < source.length; i++) {
 							if (source[i].id == id) {
@@ -23,13 +35,14 @@
 					});
 
 				function addTag(item) {
-					var tag = $('<span class="tag"/>')
+					var tag = $('<span class="badge badge-pill badge-primary"/>')
 						.text(item.toString() + ' ')
 						.data("id", item.id)
 						.append('<a>x</a>')
 						.insertBefore(input);
 					input.attr("placeholder", null);
 					select.find("option[value=" + item.id + "]").attr("selected", "selected")
+					hide_error(input);
 				}
 
 				function recreate() {
@@ -42,6 +55,13 @@
 				}
 				recreate();
 
+				input.focusout(function(e) {
+					var value = input.val().trim()
+					if (value != "") {
+						show_error(input, "Please select an existing tag, it;s not possible to add custom ones.");
+					}
+				})
+
 				input.keydown(function(e) {
 						if (e.keyCode === $.ui.keyCode.TAB && $(this).data('ui-autocomplete').menu.active)
 							e.preventDefault();
@@ -92,7 +112,7 @@
 				}
 
 				selector.click(function() { input.focus(); })
-					.delegate('.tag a', 'click', function() {
+					.delegate('.badge a', 'click', function() {
 						var id = $(this).parent().data("id");
 						for (var i = 0; i < selected.length; i++) {
 							if (selected[i] == id) {
@@ -113,13 +133,14 @@
 				}
 
 				function addTag(id, value) {
-					var tag = $('<span class="tag"/>')
+					var tag = $('<span class="badge badge-pill badge-primary"/>')
 						.text(value)
 						.data("id", id)
 						.append(' <a>x</a>')
 						.insertBefore(input);
 
 					input.attr("placeholder", null);
+					hide_error(input);
 				}
 
 				function recreate() {
@@ -147,6 +168,18 @@
 
 				result.change(readFromResult);
 
+				input.focusout(function() {
+					var item = input.val();
+					if (item.length == 0) {
+						input.data("ui-autocomplete").search("");
+					} else if (item.match(/^([a-z0-9_]+)$/)) {
+						selectItem(item);
+						recreate();
+						input.val("");
+					}
+					return true;
+				});
+
 				input.keydown(function(e) {
 						if (e.keyCode === $.ui.keyCode.TAB && $(this).data('ui-autocomplete').menu.active)
 							e.preventDefault();
@@ -159,7 +192,7 @@
 								recreate();
 								input.val("");
 							} else {
-								alert("Only lowercase alphanumeric and number names allowed.");
+								show_error(input, "Only lowercase alphanumeric and number names allowed.");
 							}
 							e.preventDefault();
 							return true;

app/public/static/topic_discard.js

@@ -0,0 +1,29 @@
+$(".topic-discard").click(function() {
+	var ele = $(this);
+	var tid = ele.attr("data-tid");
+	var discard = !ele.parent().parent().hasClass("discardtopic");
+	fetch(new Request("/api/topic_discard/?tid=" + tid +
+			"&discard=" + (discard ? "true" : "false"), {
+		method: "post",
+		credentials: "same-origin",
+		headers: {
+			"Accept": "application/json",
+			"X-CSRFToken": csrf_token,
+		},
+	})).then(function(response) {
+		response.text().then(function(txt) {
+			console.log(JSON.parse(txt));
+			if (JSON.parse(txt).discarded) {
+				ele.parent().parent().addClass("discardtopic");
+				ele.removeClass("btn-danger");
+				ele.addClass("btn-success");
+				ele.text("Show");
+			} else {
+				ele.parent().parent().removeClass("discardtopic");
+				ele.removeClass("btn-success");
+				ele.addClass("btn-danger");
+				ele.text("Discard");
+			}
+		}).catch(console.log)
+	}).catch(console.log)
+});

app/querybuilder.py

@@ -0,0 +1,130 @@
+from .models import db, PackageType, Package, ForumTopic, License, MinetestRelease, PackageRelease
+from .utils import isNo, isYes
+from sqlalchemy.sql.expression import func
+from flask import abort
+from sqlalchemy import or_
+
+class QueryBuilder:
+	title  = None
+	types  = None
+	search = None
+
+	def __init__(self, args):
+		title = "Packages"
+
+		# Get request types
+		types = args.getlist("type")
+		types = [PackageType.get(tname) for tname in types]
+		types = [type for type in types if type is not None]
+		if len(types) > 0:
+			title = ", ".join([type.value + "s" for type in types])
+
+		hide_flags = args.getlist("hide")
+
+		self.title  = title
+		self.types  = types
+		self.search = args.get("q")
+		self.random = "random" in args
+		self.lucky  = "lucky" in args
+		self.hide_nonfree = "nonfree" in hide_flags
+		self.limit  = 1 if self.lucky else None
+		self.order_by  = args.get("sort")
+		self.order_dir = args.get("order") or "desc"
+		self.protocol_version = args.get("protocol_version")
+
+		self.show_discarded = isYes(args.get("show_discarded"))
+		self.show_added = args.get("show_added")
+		if self.show_added is not None:
+			self.show_added = isYes(self.show_added)
+
+		if self.search is not None and self.search.strip() == "":
+			self.search = None
+
+	def setSortIfNone(self, name):
+		if self.order_by is None:
+			self.order_by = name
+
+	def getMinetestVersion(self):
+		if not self.protocol_version:
+			return None
+
+		self.protocol_version = int(self.protocol_version)
+		version = MinetestRelease.query.filter(MinetestRelease.protocol>=self.protocol_version).first()
+		if version is not None:
+			return version.id
+		else:
+			return 10000000
+
+	def buildPackageQuery(self):
+		query = Package.query.filter_by(soft_deleted=False, approved=True)
+
+		if len(self.types) > 0:
+			query = query.filter(Package.type.in_(self.types))
+
+		if self.search:
+			query = query.search(self.search, sort=True)
+
+		if self.random:
+			query = query.order_by(func.random())
+		else:
+			to_order = None
+			if self.order_by is None or self.order_by == "score":
+				to_order = Package.score
+			elif self.order_by == "created_at":
+				to_order = Package.created_at
+			else:
+				abort(400)
+
+			if self.order_dir == "asc":
+				to_order = db.asc(to_order)
+			elif self.order_dir == "desc":
+				to_order = db.desc(to_order)
+			else:
+				abort(400)
+
+			query = query.order_by(to_order)
+
+		if self.hide_nonfree:
+			query = query.filter(Package.license.has(License.is_foss == True))
+			query = query.filter(Package.media_license.has(License.is_foss == True))
+
+		if self.protocol_version:
+			version = self.getMinetestVersion()
+			query = query.join(Package.releases) \
+				.filter(PackageRelease.approved==True) \
+				.filter(or_(PackageRelease.min_rel_id==None, PackageRelease.min_rel_id<=version)) \
+				.filter(or_(PackageRelease.max_rel_id==None, PackageRelease.max_rel_id>=version))
+
+		if self.limit:
+			query = query.limit(self.limit)
+
+		return query
+
+	def buildTopicQuery(self, show_added=False):
+		query = ForumTopic.query
+
+		if not self.show_discarded:
+			query = query.filter_by(discarded=False)
+
+		show_added = self.show_added == True or (self.show_added is None and show_added)
+		if not show_added:
+			query = query.filter(~ db.exists().where(Package.forums==ForumTopic.topic_id))
+
+		if self.order_by is None or self.order_by == "name":
+			query = query.order_by(db.asc(ForumTopic.wip), db.asc(ForumTopic.name), db.asc(ForumTopic.title))
+		elif self.order_by == "views":
+			query = query.order_by(db.desc(ForumTopic.views))
+		elif self.order_by == "date":
+			query = query.order_by(db.asc(ForumTopic.created_at))
+			sort_by = "date"
+
+		if self.search:
+			query = query.filter(ForumTopic.title.ilike('%' + self.search + '%'))
+
+		if len(self.types) > 0:
+			query = query.filter(ForumTopic.type.in_(self.types))
+
+		if self.limit:
+			query = query.limit(self.limit)
+
+		return query

app/rediscache.py

@@ -0,0 +1,13 @@
+from . import r
+
+# This file acts as a facade between the releases code and redis,
+# and also means that the releases code avoids knowing about `app`
+
+def make_download_key(ip, package):
+	return ("{}/{}/{}").format(ip, package.author.username, package.name)
+
+def set_key(key, v):
+	r.set(key, v)
+
+def has_key(key):
+	return r.exists(key)

app/sass.py

@@ -15,8 +15,6 @@ import codecs
 from flask import *
 from scss import Scss
 
-from app import app
-
 def _convert(dir, src, dst):
 	original_wd = os.getcwd()
 	os.chdir(dir)
@@ -31,7 +29,7 @@ def _convert(dir, src, dst):
 	outfile.write(output)
 	outfile.close()
 
-def _getDirPath(originalPath, create=False):
+def _getDirPath(app, originalPath, create=False):
 	path = originalPath
 
 	if not os.path.isdir(path):
@@ -47,8 +45,8 @@ def _getDirPath(originalPath, create=False):
 
 def sass(app, inputDir='scss', outputPath='static', force=False, cacheDir="public/static"):
 	static_url_path = app.static_url_path
-	inputDir = _getDirPath(inputDir)
+	inputDir = _getDirPath(app, inputDir)
-	cacheDir = _getDirPath(cacheDir or outputPath, True)
+	cacheDir = _getDirPath(app, cacheDir or outputPath, True)
 
 	def _sass(filepath):
 		sassfile = "%s/%s.scss" % (inputDir, filepath)
@@ -63,5 +61,3 @@ def sass(app, inputDir='scss', outputPath='static', force=False, cacheDir="publi
 		return send_from_directory(cacheDir, filepath + ".css")
 
 	app.add_url_rule("/%s/<path:filepath>.css" % (outputPath), 'sass', _sass)
-
-sass(app)

app/scss/comments.scss

@@ -1,49 +1,29 @@
-.comments, .comments li {
+.img-thumbnail-1 {
-	list-style: none;
+	padding: 0px;
-	padding: 0;
+	// width: 100%;
-	margin: 0;
+	background: white;
-	border: 1px solid #444;
 }
 
 .comments {
-	border-radius: 5px;
+	list-style: none;
-	margin: 15px 0;
+	padding: 0;
-	background: #333;
 
-	.info_strip, .msg {
+	.card {
-		display: block;
+		position:relative;
-		margin: 0;
-	}
 
-	.info_strip {
+		.card-header:before {
-		padding: 0.2em 1em;
+			position: absolute;
-		border-bottom: 1px solid #444;
+			top: 11px;
-	}
+			right: 100%;
-
+			width: 0;
-	.msg {
+			height: 0;
-		padding: 1em;
+			display: block;
-		background: #222;
+			content:" ";
-	}
+			border-color: transparent;
-
+			border-style: solid solid outset;
-	.author {
+			pointer-events:none;
-		font-weight: bold;
+			border-right-color: #444;
-		float: left;
+			border-width: 14px;
-		display: inline-block;
+		}
-	}
-
-	.info_strip span {
-		float: right;
-		display: inline-block;
-		color: #bbb;
 	}
 }
-
-.comment_form {
-	margin: 1em 0;
-}
-
-.comment_form textarea {
-	min-width: 60%;
-	max-width: 100%;
-	margin: 0 0 1em 0;
-}

app/scss/components.scss

@@ -1,108 +1,3 @@
-h1 {
-	margin: 0;
-}
-
-h2, h3 {
-	margin: 5px 0;
-}
-
-a {
-	color: #0be;
-	font-weight: bold;
-	text-decoration: none;
-}
-
-a:hover {
-	color: #0df;
-	text-decoration: underline;
-}
-
-/* Containers */
-
-.box {
-	border-radius: 5px;
-	margin: 15px 0;
-	padding: 0;
-}
-
-.box > h2, .box > h3 {
-	margin: 0;
-	padding: 0.5em 0.5em 0.5em 15px;
-	border-bottom: 1px solid #444;
-}
-
-.box .box-body {
-	padding: 1em !important;
-}
-
-// .box form {
-// 	padding: 1em;
-// }
-
-.box_grey {
-	background: #333;
-	border: 1px solid #444;
-}
-
-.ul_boxes {
-	display: block;
-	margin: 0;
-	padding: 0;
-	list-style: none;
-}
-
-.ul_boxes > li {
-	padding: 0;
-	list-style: none;
-}
-
-.box_link {
-	display: block;
-	color: #ddd;
-	text-decoration: none;
-}
-
-.box_link:hover{
-	background: #3a3a3a;
-}
-
-
-/*
-	buttonset
-*/
-
-.buttonset, .buttonset li {
-	display: block;
-	margin: 0;
-	padding: 0;
-	list-style: none;
-}
-
-.buttonset {
-	margin: 15px 0;
-}
-
-.button, .buttonset li a, input[type=submit] {
-	cursor: pointer;
-}
-
-.button, .buttonset li a, input[type=submit], input[type=text],
-		input[type=password], textarea, select, .bulletselector {
-	text-align: center;
-	display: inline-block;
-	padding: 0.4em 1em;
-	background: rgba(255,255,255,0.07);
-	border: 1px solid rgba(255,255,255,0.1);
-	color: #ddd;
-	border-radius: 5px;
-	text-decoration: none;
-	font-size: 100%;
-}
-
-input[type=text], input[type=password], textarea, select, .bulletselector {
-	text-align: left;
-}
-
 .ui-autocomplete, ui-front {
 	position:absolute;
 	cursor:default;
@@ -129,74 +24,11 @@ input[type=text], input[type=password], textarea, select, .bulletselector {
 	}
 }
 
-select {
+.bulletselector {
-	min-width: 200px;
+	height: auto !important;
+	display: inline-block !important;
 }
 
-select:not([multiple]) {
-	background: linear-gradient(#444, #333);
-}
-
-.form-group {
-	padding: 8px 0;
-}
-
-.form-group label {
-	display: block;
-	vertical-align: top;
-	padding: 0 8px 8px 0;
-}
-
-.form-group input, .form-group textarea, .form-group .bulletselector {
-	display: block;
-	min-width: 100%;
-	max-width: 100%;
-}
-
-.box .form-group input, .box .form-group textarea, .form-group .bulletselector {
-	min-width: 95%;
-	max-width: 95%;
-}
-
-.form-group textarea {
-	min-height: 200px;
-}
-
-.button:hover, .buttonset li a:hover, input[type=submit]:hover {
-	background: rgba(255,255,255,0.13);
-	border: 1px solid rgba(255,255,255,0.17);
-	text-decoration: none;
-	color: #ddd;
-}
-
-.btn_green {
-	background: #363 !important;
-	border: 1px solid #473;
-}
-
-.btn_green:hover {
-	background: #474 !important;
-}
-
-.linedbuttonset a {
-	border: 1px solid #eee;
-	border-radius: 3px;
-	padding: 4px 10px;
-	margin: 0;
-	display: block;
-}
-
-.linedbuttonset {
-	display: block;
-	margin: 0;
-}
-
-.linedbuttonset li {
-	display: inline-block;
-	margin: 10px 10px 0 0;
-}
-
-
 .bulletselector input {
 	border: none;
 	border-radius: 0;
@@ -211,166 +43,14 @@ select:not([multiple]) {
 	white-space: nowrap;
 	background: transparent;
 }
-.bulletselector .tag {
+.bulletselector .badge {
-	background: #375D81;
-	border-radius: 3px;
-    -moz-border-radius: 3px;
-	color: #FFF;
 	float: left;
-	height: 15px;
+	padding: 0.4em 0.8em;
-	padding: 0.1em 0.4em 0.5em;
 	margin-right: 0.3em;
-	margin-bottom: 0.3em;
-    vertical-align: baseline;
-}
-.bulletselector .tag a {
-	color: #FFF;
-	cursor: pointer;
-}
-.bulletselector .tag a:hover {
-	color: #0099CC;
-	text-decoration: none;
 }
 
-
+.invalid-remaining {
-/* Alerts */
+	display: none;
-
-
-.alert .alert_right, .alert > form {
-	display: inline-block;
-	margin: 0;
-	position: absolute;
-	top: 0;
-	right: 0;
-	bottom: 0;
-}
-
-.alert {
-	padding: 10px;
-	position: relative;
-
-	.alert_right:not(.button) {
-		padding: 0;
-	}
-
-	.alert_right form {
-		height: 100%;
-	}
-
-	form {
-		display: inline-block;
-		margin: 0;
-		padding: 0;
-	}
-
-	input {
-		height: 100%;
-	}
-
-	input, .button {
-		margin: 0;
-		background: 0;
-		border: 0;
-		border-left: 1px solid rgba(255,255,255,0.12);
-		border-radius: 0;
-		vertical-align: middle;
-	}
-
-	input:hover, .button:hover {
-		border: 0;
-		border-left: 1px solid rgba(255,255,255,0.2);
-	}
-}
-
-#alerts {
-	list-style: none;
-	position: fixed;
-	bottom: 15px;
-	left: 0;
-	right: 0;
-	z-index: 1000;
-}
-
-#alerts .alert {
-	margin: 5px 0;
-	vertical-align: middle;
-}
-
-#alerts .close {
-	float: right;
-	color: white;
-}
-
-#alerts .close:hover {
-	color: #fff;
-}
-
-.alert-error, .button-danger {
-	background: #933 !important;
-	border: 1px solid #c44 !important;
-}
-
-.alert-warning {
-	background: #963;
-	border: 1px solid #c96;
-}
-
-.alert-primary {
-	background: #339;
-	border: 1px solid #66a;
-}
-
-.alert-success {
-	background: #161;
-	border: 1px solid #393;
-}
-
-table.fancyTable {
-	font-family: "Arial Black", Gadget, sans-serif;
-	border: 2px solid #000000;
-	background-color: #4A4A4A;
-	width: 100%;
-	text-align: center;
-	border-collapse: collapse;
-}
-table.fancyTable td, table.fancyTable th {
-	border: 1px solid #4A4A4A;
-	padding: 3px 2px;
-}
-table.fancyTable tbody td {
-	font-size: 13px;
-	color: #E6E6E6;
-}
-table.fancyTable tr:nth-child(even) {
-	background: #888888;
-}
-table.fancyTable thead {
-	background: #000000;
-	border-bottom: 3px solid #000000;
-}
-table.fancyTable thead th {
-	font-size: 15px;
-	font-weight: bold;
-	color: #E6E6E6;
-	text-align: center;
-	border-left: 2px solid #4A4A4A;
-}
-table.fancyTable thead th:first-child {
-	border-left: none;
-}
-
-table.fancyTable tfoot {
-	font-size: 12px;
-	font-weight: bold;
-	color: #E6E6E6;
-	background: #000000;
-	background: -moz-linear-gradient(top, #404040 0%, #191919 66%, #000000 100%);
-	background: -webkit-linear-gradient(top, #404040 0%, #191919 66%, #000000 100%);
-	background: linear-gradient(to bottom, #404040 0%, #191919 66%, #000000 100%);
-	border-top: 1px solid #4A4A4A;
-}
-table.fancyTable tfoot td {
-	font-size: 12px;
 }
 
 .t-mll tr td:not(:first-child) {
@@ -401,53 +81,44 @@ table.fancyTable tfoot td {
 	color: #2c2;
 }
 
-/*
+.wiptopic a:not(.btn) {
-	Aside
+	color: #7ac;
-*/
-
-.asideright {
-	float: right;
-	margin: 0 0 0 15px;
-	max-width: 300px;
 }
 
-.outsidecontainer {
+.discardtopic {
-	position: absolute;
+	text-decoration: line-through;
-	right: 102%;
+	a:not(.btn) {
-	top: 0;
+		color: #7ac;
-	width: intrinsic;           /* Safari/WebKit uses a non-standard name */
-	width: -moz-max-content;    /* Firefox/Gecko */
-	width: -webkit-max-content; /* Chrome */
-}
-
-@media (max-width: 1490px) {
-	.outsidecontainer {
-		display: none;
 	}
+	filter: brightness(0.5);
 }
 
-.flatlist, .flatlist li {
+.editor-toolbar, .editor-toolbar.fullscreen {
-	list-style: none;
+	margin-bottom: 0 !important;
-	padding: 0;
+	background-color: #444 !important;
-	margin: 0;
+	border: none !important;
+	border-radius: calc(0.25rem - 1px) calc(0.25rem - 1px) 0 0 !important;
+	border-bottom: 1px solid rgba(0, 0, 0, 0.125) !important;
 }
 
-.flatlist li {
+.editor-toolbar button {
-	display: block;
+	color: white;
 }
 
-.flatlist a {
+.editor-toolbar button.active, .editor-toolbar button:hover {
-	display: block;
+	background: #375a7f !important;
-	padding: 0.5em 20px;
+	color: white !important;
-	color: #ddd;
-	font-weight: normal;
 }
 
-.flatlist a:hover {
+.editor-toolbar.fullscreen::before, .editor-toolbar.fullscreen::after {
-	background: #444;
+	display: none !important;
-	text-decoration: none;
 }
 
-.table-topalign td {
+// .CodeMirror {
-	vertical-align: top;
+// 	background-color: #222 !important;
+// }
+
+.editor-preview-side, .editor-preview {
+	background-color: #222 !important;
+	color: white !important;
 }

app/scss/custom.scss

@@ -0,0 +1,63 @@
+@import "components.scss";
+@import "packages.scss";
+@import "packagegrid.scss";
+@import "comments.scss";
+
+.dropdown-menu {
+	margin-top: 0;
+}
+
+.dropdown:hover .dropdown-menu {
+   display: block;
+}
+
+.nav-link > img {
+	max-height: 1em;
+}
+
+#alerts {
+	display: block;
+	list-style: none;
+	position: fixed;
+	bottom: 0;
+	left:0;
+	right:0;
+	margin: 0;
+	padding:0;
+	z-index: 1000;
+}
+
+#alerts li {
+	list-style: none;
+}
+
+.jumbotron {
+	background-size: cover;
+	background-repeat: no-repeat;
+	background-position: center;
+
+	.btn-outline-secondary {
+		color: #eee;
+		border-color: #666;
+		background: rgba(102, 102, 102, 0.3);
+	}
+}
+
+.alert .btn {
+	text-decoration: none;
+}
+
+.card .table {
+	margin-bottom: 0;
+}
+
+.btn-download {
+	color: #fff;
+	background-color: #00b05c;
+	border-color: #00b05c;
+}
+
+.btn-download:focus, .btn-download.focus {
+	-webkit-box-shadow: 0 0 0 0.2rem rgba(231, 76, 60, 0.5);
+	box-shadow: 0 0 0 0.2rem rgba(231, 76, 60, 0.5);
+}

app/scss/main.scss

@@ -1,6 +0,0 @@
-@import "page.scss";
-@import "components.scss";
-@import "nav.scss";
-@import "packages.scss";
-@import "packagegrid.scss";
-@import "comments.scss";

app/scss/nav.scss

@@ -1,90 +0,0 @@
-nav {
-	margin: 0 auto 0 auto;
-	list-style: none;
-	background: #333;
-}
-
-nav .navbar-left {
-	float: left;
-}
-
-nav .navbar-left li {
-	float: left;
-}
-
-nav .navbar-right {
-	float: right;
-}
-
-nav ul {
-	margin: 0 auto 0 auto;
-	padding: 0;
-	list-style: none;
-}
-
-nav li {
-	margin: 0;
-	padding: 0;
-	list-style: none;
-	display: inline-block;
-}
-
-nav li a {
-	color: #ddd;
-	margin: 0;
-	padding: 1em 1em;
-	display: block;
-	border-left: 1px solid #444;
-}
-
-nav li a:not([href]) {
-	cursor: default;
-}
-
-nav ul li:last-child a {
-	border-right: 1px solid #444;
-}
-
-nav a:hover {
-	color: #eee;
-	background: #444;
-	text-decoration: none;
-}
-
-nav img {
-	height: 1em;
-}
-
-li.dropdown {
-	position: relative;
-	display: inline-block;
-}
-
-.dropdown-menu {
-	display: none;
-	position: absolute;
-	margin: 0;
-	padding: 0;
-	min-width:160px;
-	background: #333;
-	z-index: 1;
-	right: 0;
-	box-shadow: 0px 8px 16px 0px rgba(0,0,0,0.4);
-}
-
-.dropdown:hover ul {
-	display: block;
-}
-
-.dropdown li {
-	display: block;
-}
-
-.dropdown li a {
-	border: none;
-	border-top: 1px solid #444;
-}
-
-.dropdown li:last-child a {
-	border-bottom: 1px solid #444;
-}

app/scss/packagegrid.scss

@@ -1,27 +1,20 @@
-.packagegrid {
+.packagetile {
-	display: flex;
+	list-style: none;
-	flex-wrap: wrap;
-	flex-direction: row;
-	flex-grow: 1;
-	flex-shrink: 1;
 	padding: 0;
-	margin: 0 -7px;
+	margin: 0 7px 7px 0;
+	min-width: 250px;
 }
 
-.packagegrid li {
+li.d-flex {
-	flex: 1;
+	list-style: none;
-	display: block;
-	min-width: 300px;
-	min-height: 200px;
-	max-width: 332px;
 	padding: 0;
-	margin: 7px;
+	margin: 0;
 }
 
-.packagegrid a {
+.packagetile a {
 	display: block;
 	padding-bottom: 66.66%;
-	border-radius: 7px;
+	border-radius: 3px;
 	position: relative;
 	overflow: hidden;
 	background-size: cover;
@@ -29,10 +22,6 @@
 	background-position: center;
 }
 
-.packagegrid a:hover {
-	// box-shadow: 0px 0px 16px 6px rgba(0,0,0,0.4);
-}
-
 .packagegridscrub {
 	position: absolute;
 	top: 50%;
@@ -53,6 +42,14 @@
 
 .packagegridinfo h3 {
 	color: white;
+	font-size: 120%;
+	font-weight: bold;
+}
+
+.packagegridinfo small {
+	color: #ddd;
+	font-size: 75%;
+	font-weight: bold;
 }
 
 .packagegridinfo p {
@@ -61,15 +58,15 @@
 	font-weight: normal;
 }
 
-.packagegrid a:hover .packagegridinfo {
+.packagetile a:hover .packagegridinfo {
 	top: 0;
 }
 
-.packagegrid a:hover p {
+.packagetile a:hover p {
 	display: block;
 }
 
-.packagegrid a:hover .packagegridscrub {
+.packagetile a:hover .packagegridscrub {
 	top: 0;
 	background: rgba(0,0,0,0.8);
 }

app/scss/packages.scss

@@ -37,34 +37,6 @@
 	left: 15px;
 }
 
-.sidebar_container {
-	display: block;
-	position: relative;
-	padding: 0;
-	margin: 0;
-}
-
-.sidebar_container .right, .sidebar_container .left{
-	position: absolute;
-	display: block;
-	top: 10px;
-	margin-top: 0;
-}
-
-.sidebar_container .right {
-	right: 0;
-	width: 280px;
-}
-
-.sidebar_container .left {
-	right: 295px;
-	left: 0;
-}
-
-.sidebar_container .right > *:first-child, .sidebar_container .left > *:first-child {
-	margin-top: 0;
-}
-
 .package-short-large {
 	font-size: 120%;
 }

app/scss/page.scss

@@ -1,56 +0,0 @@
-html, body {
-	font-family: "Arial", sans-serif;
-	background: #222;
-	color: #ddd;
-	padding: 0;
-	margin: 0;
-}
-
-.container, main, #alerts, footer {
-	width: 90%;
-	max-width: 1024px;
-	margin: auto;
-	padding: 0;
-	display: block;
-}
-
-main {
-	padding: 7px 0;
-	position: relative;
-	box-sizing: border-box;
-}
-
-.clearboth {
-	clear: both;
-}
-
-header h1, header p, header form {
-	padding: 0 5px;
-	margin-left: 0;
-}
-
-header {
-	padding: 30px;
-	background: #258;
-}
-
-header p {
-	max-width: 400px;
-}
-
-header input {
-	margin: 3px;
-}
-
-footer {
-	color: #999;
-	padding: 30px 0;
-}
-
-footer a {
-	color: #aaa;
-}
-
-footer a:hover {
-	color: #bbb;
-}

app/tasks/__init__.py

@@ -16,8 +16,9 @@
 
 
 import flask
-from flask.ext.sqlalchemy import SQLAlchemy
+from flask_sqlalchemy import SQLAlchemy
 from celery import Celery
+from celery.schedules import crontab
 from app import app
 from app.models import *
 
@@ -64,4 +65,16 @@ def make_celery(app):
 
 celery = make_celery(app)
 
-from . import importtasks, forumtasks, emails
+CELERYBEAT_SCHEDULE = {
+	'topic_list_import': {
+		'task': 'app.tasks.forumtasks.importTopicList',
+		'schedule': crontab(minute=1, hour=1),
+	},
+	'package_score_update': {
+		'task': 'app.tasks.pkgtasks.updatePackageScores',
+		'schedule': crontab(minute=10, hour=1),
+	}
+}
+celery.conf.beat_schedule = CELERYBEAT_SCHEDULE
+
+from . import importtasks, forumtasks, emails, pkgtasks

app/tasks/emails.py

@@ -15,14 +15,36 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-from flask import *
+from flask import render_template, url_for
 from flask_mail import Message
 from app import mail
 from app.tasks import celery
 
 @celery.task()
 def sendVerifyEmail(newEmail, token):
-    msg = Message("Verify email address", recipients=[newEmail])
+	print("Sending verify email!")
-    msg.body = "This is a verification email!"
+	msg = Message("Verify email address", recipients=[newEmail])
-    msg.html = render_template("emails/verify.html", token=token)
+
-    mail.send(msg)
+	msg.body = """
+			This email has been sent to you because someone (hopefully you)
+			has entered your email address as a user's email.
+
+			If it wasn't you, then just delete this email.
+
+			If this was you, then please click this link to verify the address:
+
+			{}
+		""".format(url_for('users.verify_email', token=token, _external=True))
+
+	msg.html = render_template("emails/verify.html", token=token)
+	mail.send(msg)
+
+@celery.task()
+def sendEmailRaw(to, subject, text, html):
+	from flask_mail import Message
+	msg = Message(subject, recipients=to)
+
+	msg.body = text or html
+	html = html or text
+	msg.html = render_template("emails/base.html", subject=subject, content=html)
+	mail.send(msg)

app/tasks/forumtasks.py

@@ -15,17 +15,18 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-import flask, json
+import flask, json, re
-from flask.ext.sqlalchemy import SQLAlchemy
+from flask_sqlalchemy import SQLAlchemy
 from app import app
 from app.models import *
 from app.tasks import celery
-from .phpbbparser import getProfile
+from .phpbbparser import getProfile, getTopicsFromForum
 import urllib.request
 from urllib.parse import urlparse, quote_plus
 
 @celery.task()
-def checkForumAccount(username, token=None):
+def checkForumAccount(username, forceNoSave=False):
+	print("Checking " + username)
 	try:
 		profile = getProfile("https://forum.minetest.net", username)
 	except OSError:
@@ -47,75 +48,127 @@ def checkForumAccount(username, token=None):
 		user.github_username = github_username
 		needsSaving = True
 
+	pic = profile.avatar
+	if pic and "http" in pic:
+		pic = None
+
+	needsSaving = needsSaving or pic != user.profile_pic
+	if pic:
+		user.profile_pic = "https://forum.minetest.net/" + pic
+	else:
+		user.profile_pic = None
+
 	# Save
-	if needsSaving:
+	if needsSaving and not forceNoSave:
 		db.session.commit()
 
-@celery.task()
+	return needsSaving
-def importUsersFromModList():
-	contents = urllib.request.urlopen("http://krock-works.16mb.com/MTstuff/modList.php").read().decode("utf-8")
-	list = json.loads(contents)
-	found = {}
-	imported = []
-
-	for user in User.query.all():
-		found[user.username] = True
-		if user.forums_username is not None:
-			found[user.forums_username] = True
-
-	for x in list:
-		author = x.get("author")
-		if author is not None and not author in found:
-			user = User(author)
-			user.forums_username = author
-			imported.append(author)
-			found[author] = True
-			db.session.add(user)
-
-	db.session.commit()
-	for author in found:
-		checkForumAccount.delay(author, None)
-
-
-BANNED_NAMES = ["mod", "game", "old", "outdated", "wip", "api"]
-ALLOWED_TYPES = [1, 2, 6]
 
 @celery.task()
-def importKrocksModList():
+def checkAllForumAccounts(forceNoSave=False):
-	contents = urllib.request.urlopen("http://krock-works.16mb.com/MTstuff/modList.php").read().decode("utf-8")
+	needsSaving = False
-	list = json.loads(contents)
+	query = User.query.filter(User.forums_username.isnot(None))
+	for user in query.all():
+		needsSaving = checkForumAccount(user.username) or needsSaving
+
+	if needsSaving and not forceNoSave:
+		db.session.commit()
+
+	return needsSaving
+
+
+regex_tag    = re.compile(r"\[([a-z0-9_]+)\]")
+BANNED_NAMES = ["mod", "game", "old", "outdated", "wip", "api", "beta", "alpha", "git"]
+def getNameFromTaglist(taglist):
+	for tag in reversed(regex_tag.findall(taglist)):
+		if len(tag) < 30 and not tag in BANNED_NAMES and \
+				not re.match(r"^[a-z]?[0-9]+$", tag):
+			return tag
+
+	return None
+
+regex_title = re.compile(r"^((?:\[[^\]]+\] *)*)([^\[]+) *((?:\[[^\]]+\] *)*)[^\[]*$")
+def parseTitle(title):
+	m = regex_title.match(title)
+	if m is None:
+		print("Invalid title format: " + title)
+		return title, getNameFromTaglist(title)
+	else:
+		return m.group(2).strip(), getNameFromTaglist(m.group(3))
+
+def getLinksFromModSearch():
+	links = {}
+
+	try:
+		contents = urllib.request.urlopen("https://krock-works.uk.to/minetest/modList.php").read().decode("utf-8")
+		for x in json.loads(contents):
+			try:
+				link = x.get("link")
+				if link is not None:
+					links[int(x["topicId"])] = link
+			except ValueError:
+				pass
+
+	except urllib.error.URLError:
+		print("Unable to open krocks mod search!")
+		return links
+
+	return links
+
+@celery.task()
+def importTopicList():
+	links_by_id = getLinksFromModSearch()
+
+	info_by_id = {}
+	getTopicsFromForum(11, out=info_by_id, extra={ 'type': PackageType.MOD,  'wip': False })
+	getTopicsFromForum(9,  out=info_by_id, extra={ 'type': PackageType.MOD,  'wip': True  })
+	getTopicsFromForum(15, out=info_by_id, extra={ 'type': PackageType.GAME, 'wip': False })
+	getTopicsFromForum(50, out=info_by_id, extra={ 'type': PackageType.GAME, 'wip': True  })
+
+	# Caches
 	username_to_user = {}
+	topics_by_id     = {}
+	for topic in ForumTopic.query.all():
+		topics_by_id[topic.topic_id] = topic
 
-	KrockForumTopic.query.delete()
+	# Create or update
+	for info in info_by_id.values():
+		id = int(info["id"])
 
-	for x in list:
+		# Get author
-		type = int(x["type"])
+		username = info["author"]
-		if not type in ALLOWED_TYPES:
-			continue
-
-		username = x["author"]
 		user = username_to_user.get(username)
 		if user is None:
 			user = User.query.filter_by(forums_username=username).first()
-			assert(user is not None)
+			if user is None:
+				print(username + " not found!")
+				user = User(username)
+				user.forums_username = username
+				db.session.add(user)
 			username_to_user[username] = user
 
-		import re
+		# Get / add row
-		tags = re.findall("\[([a-z0-9_]+)\]", x["title"])
+		topic = topics_by_id.get(id)
-		name = None
+		if topic is None:
-		for tag in reversed(tags):
+			topic = ForumTopic()
-			if len(tag) < 30 and not tag in BANNED_NAMES and \
+			db.session.add(topic)
-					not re.match("^([a-z][0-9]+)$", tag):
-				name = tag
-				break
 
-		topic = KrockForumTopic()
+		# Parse title
-		topic.topic_id  = x["topicId"]
+		title, name = parseTitle(info["title"])
-		topic.author_id = user.id
+
-		topic.ttype     = type
+		# Get link
-		topic.title     = x["title"]
+		link = links_by_id.get(id)
-		topic.name      = name
+
-		topic.link      = x.get("link")
+		# Fill row
-		db.session.add(topic)
+		topic.topic_id   = int(id)
+		topic.author     = user
+		topic.type       = info["type"]
+		topic.title      = title
+		topic.name       = name
+		topic.link       = link
+		topic.wip        = info["wip"]
+		topic.posts      = int(info["posts"])
+		topic.views      = int(info["views"])
+		topic.created_at = info["date"]
 
 	db.session.commit()

app/tasks/importtasks.py

@@ -15,20 +15,28 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-import flask, json, os, git, tempfile, shutil
+import flask, json, os, git, tempfile, shutil, gitdb
 from git import GitCommandError
-from flask.ext.sqlalchemy import SQLAlchemy
+from git_archive_all import GitArchiver
+from flask_sqlalchemy import SQLAlchemy
 from urllib.error import HTTPError
 import urllib.request
 from urllib.parse import urlparse, quote_plus, urlsplit
+from zipfile import ZipFile
+
 from app import app
 from app.models import *
 from app.tasks import celery, TaskError
 from app.utils import randomString
-
+from .minetestcheck import build_tree, MinetestCheckError, ContentType
+from .minetestcheck.config import parse_conf
 
 class GithubURLMaker:
 	def __init__(self, url):
+		self.baseUrl = None
+		self.user = None
+		self.repo = None
+
 		# Rewrite path
 		import re
 		m = re.search("^\/([^\/]+)\/([^\/]+)\/?$", url.path)
@@ -51,6 +59,9 @@ class GithubURLMaker:
 	def getScreenshotURL(self):
 		return self.baseUrl + "/screenshot.png"
 
+	def getModConfURL(self):
+		return self.baseUrl + "/mod.conf"
+
 	def getCommitsURL(self, branch):
 		return "https://api.github.com/repos/{}/{}/commits?sha={}" \
 				.format(self.user, self.repo, urllib.parse.quote_plus(branch))
@@ -66,7 +77,7 @@ def getKrockList():
 	global krock_list_cache_by_name
 
 	if krock_list_cache is None:
-		contents = urllib.request.urlopen("http://krock-works.16mb.com/MTstuff/modList.php").read().decode("utf-8")
+		contents = urllib.request.urlopen("https://krock-works.uk.to/minetest/modList.php").read().decode("utf-8")
 		list = json.loads(contents)
 
 		def h(x):
@@ -119,187 +130,44 @@ def findModInfo(author, name, link):
 
 	return None
 
-
-def parseConf(string):
-	retval = {}
-	for line in string.split("\n"):
-		idx = line.find("=")
-		if idx > 0:
-			key   = line[:idx].strip()
-			value = line[idx+1:].strip()
-			retval[key] = value
-
-	return retval
-
-
-class PackageTreeNode:
-	def __init__(self, baseDir, author=None, repo=None, name=None):
-		print("Scanning " + baseDir)
-		self.baseDir  = baseDir
-		self.author   = author
-		self.name	 = name
-		self.repo	 = repo
-		self.meta	 = None
-		self.children = []
-
-		# Detect type
-		type = None
-		is_modpack = False
-		if os.path.isfile(baseDir + "/game.conf"):
-			type = PackageType.GAME
-		elif os.path.isfile(baseDir + "/init.lua"):
-			type = PackageType.MOD
-		elif os.path.isfile(baseDir + "/modpack.txt"):
-			type = PackageType.MOD
-			is_modpack = True
-		elif os.path.isdir(baseDir + "/mods"):
-			type = PackageType.GAME
-		elif os.listdir(baseDir) == []:
-			# probably a submodule
-			return
-		else:
-			raise TaskError("Unable to detect package type!")
-
-		self.type = type
-		self.readMetaFiles()
-
-		if self.type == PackageType.GAME:
-			self.addChildrenFromModDir(baseDir + "/mods")
-		elif is_modpack:
-			self.addChildrenFromModDir(baseDir)
-
-
-	def readMetaFiles(self):
-		result = {}
-
-		# .conf file
-		try:
-			with open(self.baseDir + "/mod.conf", "r") as myfile:
-				conf = parseConf(myfile.read())
-				for key in ["name", "description", "title", "depends", "optional_depends"]:
-					try:
-						result[key] = conf[key]
-					except KeyError:
-						pass
-		except IOError:
-			print("description.txt does not exist!")
-
-		# description.txt
-		if not "description" in result:
-			try:
-				with open(self.baseDir + "/description.txt", "r") as myfile:
-					result["description"] = myfile.read()
-			except IOError:
-				print("description.txt does not exist!")
-
-		# depends.txt
-		import re
-		pattern = re.compile("^([a-z0-9_]+)\??$")
-		if not "depends" in result and not "optional_depends" in result:
-			try:
-				with open(self.baseDir + "/depends.txt", "r") as myfile:
-					contents = myfile.read()
-					soft = []
-					hard = []
-					for line in contents.split("\n"):
-						line = line.strip()
-						if pattern.match(line):
-							if line[len(line) - 1] == "?":
-								soft.append( line[:-1])
-							else:
-								hard.append(line)
-
-					result["depends"] = hard
-					result["optional_depends"] = soft
-
-			except IOError:
-				print("depends.txt does not exist!")
-
-		else:
-			if "depends" in result:
-				result["depends"] = [x.strip() for x in result["depends"].split(",")]
-			if "optional_depends" in result:
-				result["optional_depends"] = [x.strip() for x in result["optional_depends"].split(",")]
-
-
-		# Calculate Title
-		if "name" in result and not "title" in result:
-			result["title"] = result["name"].replace("_", " ").title()
-
-		# Calculate short description
-		if "description" in result:
-			desc = result["description"]
-			idx = desc.find(".") + 1
-			cutIdx = min(len(desc), 200 if idx < 5 else idx)
-			result["short_description"] = desc[:cutIdx]
-
-		# Get forum ID
-		info = findModInfo(self.author, result.get("name"), self.repo)
-		if info is not None:
-			result["forumId"] = info.get("topicId")
-
-		if "name" in result:
-			self.name = result["name"]
-			del result["name"]
-
-		self.meta = result
-
-	def addChildrenFromModDir(self, dir):
-		for entry in next(os.walk(dir))[1]:
-			path = dir + "/" + entry
-			if not entry.startswith('.') and os.path.isdir(path):
-				self.children.append(PackageTreeNode(path, name=entry))
-
-
-	def fold(self, attr, key=None, acc=None):
-		if acc is None:
-			acc = set()
-
-		if self.meta is None:
-			return acc
-
-		at = getattr(self, attr)
-		value = at if key is None else at.get(key)
-
-		if isinstance(value, list):
-			acc |= set(value)
-		elif value is not None:
-			acc.add(value)
-
-		for child in self.children:
-			child.fold(attr, key, acc)
-
-		return acc
-
-	def get(self, key):
-		return self.meta.get(key)
-
 def generateGitURL(urlstr):
 	scheme, netloc, path, query, frag = urlsplit(urlstr)
 
 	return "http://:@" + netloc + path + query
 
+
+def getTempDir():
+	return os.path.join(tempfile.gettempdir(), randomString(10))
+
+
 # Clones a repo from an unvalidated URL.
 # Returns a tuple of path and repo on sucess.
 # Throws `TaskError` on failure.
 # Caller is responsible for deleting returned directory.
 def cloneRepo(urlstr, ref=None, recursive=False):
-	gitDir = tempfile.gettempdir() + "/" + randomString(10)
+	gitDir = getTempDir()
 
 	err = None
 	try:
 		gitUrl = generateGitURL(urlstr)
 		print("Cloning from " + gitUrl)
-		repo = git.Repo.clone_from(gitUrl, gitDir, \
-				progress=None, env=None, depth=1, recursive=recursive, kill_after_timeout=15)
 
-		if ref is not None:
+		if ref is None:
-			repo.create_head("myhead", ref).checkout()
+			repo = git.Repo.clone_from(gitUrl, gitDir, \
+					progress=None, env=None, depth=1, recursive=recursive, kill_after_timeout=15)
+		else:
+			repo = git.Repo.clone_from(gitUrl, gitDir, \
+					progress=None, env=None, depth=1, recursive=recursive, kill_after_timeout=15, b=ref)
+
 		return gitDir, repo
+
 	except GitCommandError as e:
 		# This is needed to stop the backtrace being weird
 		err = e.stderr
 
+	except gitdb.exc.BadName as e:
+		err = "Unable to find the reference " + (ref or "?") + "\n" + e.stderr
+
 	raise TaskError(err.replace("stderr: ", "") \
 			.replace("Cloning into '" + gitDir + "'...", "") \
 			.strip())
@@ -307,7 +175,12 @@ def cloneRepo(urlstr, ref=None, recursive=False):
 @celery.task()
 def getMeta(urlstr, author):
 	gitDir, _ = cloneRepo(urlstr, recursive=True)
-	tree = PackageTreeNode(gitDir, author=author, repo=urlstr)
+
+	try:
+		tree = build_tree(gitDir, author=author, repo=urlstr)
+	except MinetestCheckError as err:
+		raise TaskError(str(err))
+
 	shutil.rmtree(gitDir)
 
 	result = {}
@@ -338,20 +211,57 @@ def makeVCSReleaseFromGithub(id, branch, release, url):
 		raise TaskError("Invalid github repo URL")
 
 	commitsURL = urlmaker.getCommitsURL(branch)
-	contents = urllib.request.urlopen(commitsURL).read().decode("utf-8")
+	try:
-	commits = json.loads(contents)
+		contents = urllib.request.urlopen(commitsURL).read().decode("utf-8")
+		commits = json.loads(contents)
+	except HTTPError:
+		raise TaskError("Unable to get commits for Github repository. Either the repository or reference doesn't exist.")
 
 	if len(commits) == 0 or not "sha" in commits[0]:
 		raise TaskError("No commits found")
 
-	release.url = urlmaker.getCommitDownload(commits[0]["sha"])
+	release.url          = urlmaker.getCommitDownload(commits[0]["sha"])
-	print(release.url)
+	release.task_id     = None
-	release.task_id = None
+	release.commit_hash = commits[0]["sha"]
+	release.approve(release.package.author)
 	db.session.commit()
 
 	return release.url
 
 
+@celery.task(bind=True)
+def checkZipRelease(self, id, path):
+	release = PackageRelease.query.get(id)
+	if release is None:
+		raise TaskError("No such release!")
+	elif release.package is None:
+		raise TaskError("No package attached to release")
+
+	temp = getTempDir()
+	try:
+		with ZipFile(path, 'r') as zip_ref:
+			zip_ref.extractall(temp)
+
+		try:
+			tree = build_tree(temp, expected_type=ContentType[release.package.type.name], \
+				author=release.package.author.username, name=release.package.name)
+		except MinetestCheckError as err:
+			if "Fails validation" not in release.title:
+				release.title += " (Fails validation)"
+
+			release.task_id = self.request.id
+			release.approved = False
+			db.session.commit()
+
+			raise TaskError(str(err))
+
+		release.task_id = None
+		release.approve(release.package.author)
+		db.session.commit()
+
+	finally:
+		shutil.rmtree(temp)
+
 
 @celery.task()
 def makeVCSRelease(id, branch):
@@ -361,27 +271,37 @@ def makeVCSRelease(id, branch):
 	elif release.package is None:
 		raise TaskError("No package attached to release")
 
-	urlmaker = None
+	# url = urlparse(release.package.repo)
-	url = urlparse(release.package.repo)
+	# if url.netloc == "github.com":
-	if url.netloc == "github.com":
+	# 	return makeVCSReleaseFromGithub(id, branch, release, url)
-		return makeVCSReleaseFromGithub(id, branch, release, url)
-	else:
-		gitDir, repo = cloneRepo(release.package.repo, ref=branch, recursive=True)
 
-		try:
+	gitDir, repo = cloneRepo(release.package.repo, ref=branch, recursive=True)
-			filename = randomString(10) + ".zip"
-			destPath = os.path.join("app/public/uploads", filename)
-			with open(destPath, "wb") as fp:
-				repo.archive(fp)
 
-			release.url = "/uploads/" + filename
+	try:
-			print(release.url)
+		tree = build_tree(gitDir, expected_type=ContentType[release.package.type.name], \
-			release.task_id = None
+			author=release.package.author.username, name=release.package.name)
-			db.session.commit()
+	except MinetestCheckError as err:
+		raise TaskError(str(err))
 
-			return release.url
+	try:
-		finally:
+		filename = randomString(10) + ".zip"
-			shutil.rmtree(gitDir)
+		destPath = os.path.join(app.config["UPLOAD_DIR"], filename)
+
+		assert(not os.path.isfile(destPath))
+		archiver = GitArchiver(force_sub=True, main_repo_abspath=gitDir)
+		archiver.create(destPath)
+		assert(os.path.isfile(destPath))
+
+		release.url         = "/uploads/" + filename
+		release.task_id     = None
+		release.commit_hash = repo.head.object.hexsha
+		release.approve(release.package.author)
+		print(release.url)
+		db.session.commit()
+
+		return release.url
+	finally:
+		shutil.rmtree(gitDir)
 
 @celery.task()
 def importRepoScreenshot(id):
@@ -403,7 +323,7 @@ def importRepoScreenshot(id):
 			sourcePath = gitDir + "/screenshot." + ext
 			if os.path.isfile(sourcePath):
 				filename = randomString(10) + "." + ext
-				destPath = os.path.join("app/public/uploads", filename)
+				destPath = os.path.join(app.config["UPLOAD_DIR"], filename)
 				shutil.copyfile(sourcePath, destPath)
 
 				ss = PackageScreenshot()
@@ -440,7 +360,7 @@ def getDepends(package):
 	#
 	try:
 		contents = urllib.request.urlopen(urlmaker.getModConfURL()).read().decode("utf-8")
-		conf = parseConf(contents)
+		conf = parse_conf(contents)
 		for key in ["depends", "optional_depends"]:
 			try:
 				result[key] = conf[key]

app/tasks/minetestcheck/__init__.py

@@ -0,0 +1,48 @@
+from enum import Enum
+
+class MinetestCheckError(Exception):
+	def __init__(self, value):
+		self.value = value
+	def __str__(self):
+		return repr("Error validating package: " + self.value)
+
+class ContentType(Enum):
+	UNKNOWN = "unknown"
+	MOD = "mod"
+	MODPACK = "modpack"
+	GAME = "game"
+	TXP = "texture pack"
+
+	def isModLike(self):
+		return self == ContentType.MOD or self == ContentType.MODPACK
+
+	def validate_same(self, other):
+		"""
+		Whether or not `other` is an acceptable type for this
+		"""
+		assert(other)
+
+		if self == ContentType.MOD:
+			if not other.isModLike():
+				raise MinetestCheckError("expected a mod or modpack, found " + other.value)
+
+		elif self == ContentType.TXP:
+			if other != ContentType.UNKNOWN and other != ContentType.TXP:
+				raise MinetestCheckError("expected a " + self.value + ", found a " + other.value)
+
+		elif other != self:
+			raise MinetestCheckError("expected a " + self.value + ", found a " + other.value)
+
+
+from .tree import PackageTreeNode, get_base_dir
+
+def build_tree(path, expected_type=None, author=None, repo=None, name=None):
+	path = get_base_dir(path)
+
+	root = PackageTreeNode(path, "/", author=author, repo=repo, name=name)
+	assert(root)
+
+	if expected_type:
+		expected_type.validate_same(root.type)
+
+	return root

app/tasks/minetestcheck/config.py

@@ -0,0 +1,10 @@
+def parse_conf(string):
+	retval = {}
+	for line in string.split("\n"):
+		idx = line.find("=")
+		if idx > 0:
+			key   = line[:idx].strip()
+			value = line[idx+1:].strip()
+			retval[key] = value
+
+	return retval

app/tasks/minetestcheck/tree.py

@@ -0,0 +1,162 @@
+import os
+from . import MinetestCheckError, ContentType
+from .config import parse_conf
+
+def get_base_dir(path):
+	if not os.path.isdir(path):
+		raise IOError("Expected dir")
+
+	root, subdirs, files = next(os.walk(path))
+	if len(subdirs) == 1 and len(files) == 0:
+		return get_base_dir(path + "/" + subdirs[0])
+	else:
+		return path
+
+
+def detect_type(path):
+	if os.path.isfile(path + "/game.conf"):
+		return ContentType.GAME
+	elif os.path.isfile(path + "/init.lua"):
+		return ContentType.MOD
+	elif os.path.isfile(path + "/modpack.txt") or \
+			os.path.isfile(path + "/modpack.conf"):
+		return ContentType.MODPACK
+	elif os.path.isdir(path + "/mods"):
+		return ContentType.GAME
+	elif os.path.isfile(path + "/texture_pack.conf"):
+		return ContentType.TXP
+	else:
+		return ContentType.UNKNOWN
+
+
+class PackageTreeNode:
+	def __init__(self, baseDir, relative, author=None, repo=None, name=None):
+		print(baseDir)
+		self.baseDir  = baseDir
+		self.relative = relative
+		self.author   = author
+		self.name	 = name
+		self.repo	 = repo
+		self.meta	 = None
+		self.children = []
+
+		# Detect type
+		self.type = detect_type(baseDir)
+		self.read_meta()
+
+		if self.type == ContentType.GAME:
+			if not os.path.isdir(baseDir + "/mods"):
+				raise MinetestCheckError(("game at {} does not have a mods/ folder").format(self.relative))
+			self.add_children_from_mod_dir(baseDir + "/mods")
+		elif self.type == ContentType.MODPACK:
+			self.add_children_from_mod_dir(baseDir)
+
+
+	def read_meta(self):
+		result = {}
+
+		# .conf file
+		try:
+			with open(self.baseDir + "/mod.conf", "r") as myfile:
+				conf = parse_conf(myfile.read())
+				for key in ["name", "description", "title", "depends", "optional_depends"]:
+					try:
+						result[key] = conf[key]
+					except KeyError:
+						pass
+		except IOError:
+			pass
+
+		# description.txt
+		if not "description" in result:
+			try:
+				with open(self.baseDir + "/description.txt", "r") as myfile:
+					result["description"] = myfile.read()
+			except IOError:
+				pass
+
+		# depends.txt
+		import re
+		pattern = re.compile("^([a-z0-9_]+)\??$")
+		if not "depends" in result and not "optional_depends" in result:
+			try:
+				with open(self.baseDir + "/depends.txt", "r") as myfile:
+					contents = myfile.read()
+					soft = []
+					hard = []
+					for line in contents.split("\n"):
+						line = line.strip()
+						if pattern.match(line):
+							if line[len(line) - 1] == "?":
+								soft.append( line[:-1])
+							else:
+								hard.append(line)
+
+					result["depends"] = hard
+					result["optional_depends"] = soft
+
+			except IOError:
+				pass
+
+		else:
+			if "depends" in result:
+				result["depends"] = [x.strip() for x in result["depends"].split(",")]
+			if "optional_depends" in result:
+				result["optional_depends"] = [x.strip() for x in result["optional_depends"].split(",")]
+
+
+		# Calculate Title
+		if "name" in result and not "title" in result:
+			result["title"] = result["name"].replace("_", " ").title()
+
+		# Calculate short description
+		if "description" in result:
+			desc = result["description"]
+			idx = desc.find(".") + 1
+			cutIdx = min(len(desc), 200 if idx < 5 else idx)
+			result["short_description"] = desc[:cutIdx]
+
+		if "name" in result:
+			self.name = result["name"]
+			del result["name"]
+
+		self.meta = result
+
+	def add_children_from_mod_dir(self, dir):
+		for entry in next(os.walk(dir))[1]:
+			path = os.path.join(dir, entry)
+			if not entry.startswith('.') and os.path.isdir(path):
+				child = PackageTreeNode(path, self.relative + entry + "/", name=entry)
+				if not child.type.isModLike():
+					raise MinetestCheckError(("Expecting mod or modpack, found {} at {} inside {}") \
+							.format(child.type.value, child.relative, self.type.value))
+
+				self.children.append(child)
+
+
+	def fold(self, attr, key=None, acc=None):
+		if acc is None:
+			acc = set()
+
+		if self.meta is None:
+			return acc
+
+		at = getattr(self, attr)
+		value = at if key is None else at.get(key)
+
+		if isinstance(value, list):
+			acc |= set(value)
+		elif value is not None:
+			acc.add(value)
+
+		for child in self.children:
+			child.fold(attr, key, acc)
+
+		return acc
+
+	def get(self, key):
+		return self.meta.get(key)
+
+	def validate(self):
+		for child in self.children:
+			child.validate()

app/tasks/phpbbparser.py

@@ -5,6 +5,7 @@
 import urllib, socket
 from bs4 import *
 from urllib.parse import urljoin
+from datetime import datetime
 import urllib.request
 import os.path
 import time, re
@@ -14,8 +15,9 @@ def urlEncodeNonAscii(b):
 
 class Profile:
 	def __init__(self, username):
-		self.username = username
+		self.username   = username
-		self.signature = ""
+		self.signature  = ""
+		self.avatar     = None
 		self.properties = {}
 
 	def set(self, key, value):
@@ -32,6 +34,11 @@ def __extract_properties(profile, soup):
 	if el is None:
 		return None
 
+	res1 = el.find_all("dl")
+	imgs = res1[0].find_all("img")
+	if len(imgs) == 1:
+		profile.avatar = imgs[0]["src"]
+
 	res = el.find_all("dl", class_ = "left-box details")
 	if len(res) != 1:
 		return None
@@ -77,3 +84,72 @@ def getProfile(url, username):
 		__extract_properties(profile, soup)
 
 		return profile
+
+
+regex_id = re.compile(r"^.*t=([0-9]+).*$")
+
+def parseForumListPage(id, page, out, extra=None):
+	num_per_page = 30
+	start = page*num_per_page+1
+	print(" - Fetching page {} (topics {}-{})".format(page, start, start+num_per_page))
+
+	url = "https://forum.minetest.net/viewforum.php?f=" + str(id) + "&start=" + str(start)
+	r = urllib.request.urlopen(url).read().decode("utf-8")
+	soup = BeautifulSoup(r, "html.parser")
+
+	for row in soup.find_all("li", class_="row"):
+		classes = row.get("class")
+		if "sticky" in classes or "announce" in classes or "global-announce" in classes:
+			continue
+
+		topic = row.find("dl")
+
+		# Link info
+		link   = topic.find(class_="topictitle")
+		id	   = regex_id.match(link.get("href")).group(1)
+		title  = link.find(text=True)
+
+		# Date
+		left   = topic.find("dt")
+		date   = left.get_text().split("»")[1].strip()
+		date   = datetime.strptime(date, "%a %b %d, %Y %H:%M")
+		author = left.find_all("a")[-1].get_text().strip()
+
+		# Get counts
+		posts  = topic.find(class_="posts").find(text=True)
+		views  = topic.find(class_="views").find(text=True)
+
+		if id in out:
+			print("   - got {} again, title: {}".format(id, title))
+			assert title == out[id]['title']
+			return False
+
+		row = {
+			"id"    : id,
+			"title" : title,
+			"author": author,
+			"posts" : posts,
+			"views" : views,
+			"date"  : date
+		}
+
+		if extra is not None:
+			for key, value in extra.items():
+				row[key] = value
+
+		out[id] = row
+
+	return True
+
+def getTopicsFromForum(id, out={}, extra=None):
+	print("Fetching all topics from forum {}".format(id))
+	page = 0
+	while parseForumListPage(id, page, out, extra):
+		page = page + 1
+
+	return out
+
+def dumpTitlesToFile(topics, path):
+	with open(path, "w") as out_file:
+		for topic in topics.values():
+			out_file.write(topic["title"] + "\n")

app/tasks/pkgtasks.py

@@ -0,0 +1,23 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from app.models import Package
+from app.tasks import celery
+
+@celery.task()
+def updatePackageScores():
+	Package.query.update({ "score": Package.score * 0.8 })

app/template_filters.py

@@ -0,0 +1,22 @@
+from . import app
+from urllib.parse import urlparse
+
+@app.context_processor
+def inject_debug():
+    return dict(debug=app.debug)
+
+@app.template_filter()
+def throw(err):
+	raise Exception(err)
+
+@app.template_filter()
+def domain(url):
+	return urlparse(url).netloc
+
+@app.template_filter()
+def date(value):
+    return value.strftime("%Y-%m-%d")
+
+@app.template_filter()
+def datetime(value):
+    return value.strftime("%Y-%m-%d %H:%M") + " UTC"

app/templates/admin/licenses/edit.html

@@ -0,0 +1,25 @@
+{% extends "base.html" %}
+
+{% block title %}
+	{% if license %}
+		Edit {{ license.name }}
+	{% else %}
+		New license
+	{% endif %}
+{% endblock %}
+
+{% block content %}
+	<p>
+		<a href="{{ url_for('admin.license_list') }}">Back to list</a> |
+		<a href="{{ url_for('admin.create_edit_license') }}">New License</a>
+	</p>
+
+	{% from "macros/forms.html" import render_field, render_submit_field %}
+	<form method="POST" action="" enctype="multipart/form-data">
+		{{ form.hidden_tag() }}
+
+		{{ render_field(form.name) }}
+		{{ render_field(form.is_foss) }}
+		{{ render_submit_field(form.submit) }}
+	</form>
+{% endblock %}

app/templates/admin/licenses/list.html

@@ -0,0 +1,16 @@
+{% extends "base.html" %}
+
+{% block title %}
+Licenses
+{% endblock %}
+
+{% block content %}
+	<p>
+		<a href="{{ url_for('admin.create_edit_license') }}">New License</a>
+	</p>
+	<ul>
+		{% for l in licenses %}
+			<li><a href="{{ url_for('admin.create_edit_license', name=l.name) }}">{{ l.name }}</a> [{{ l.is_foss and "Free" or "Non-free"}}]</li>
+		{% endfor %}
+	</ul>
+{% endblock %}

app/templates/admin/list.html

@@ -6,31 +6,37 @@
 
 {% block content %}
 	<ul>
-		<li><a href="{{ url_for('user_list_page') }}">User list</a></li>
+		<li><a href="{{ url_for('users.list_all') }}">User list</a></li>
-		<li><a href="{{ url_for('tag_list_page') }}">Tag Editor</a></li>
+		<li><a href="{{ url_for('admin.tag_list') }}">Tag Editor</a></li>
-		<li><a href="{{ url_for('switch_user_page') }}">Sign in as another user</a></li>
+		<li><a href="{{ url_for('admin.license_list') }}">License Editor</a></li>
+		<li><a href="{{ url_for('admin.version_list') }}">Version Editor</a></li>
+		<li><a href="{{ url_for('admin.switch_user') }}">Sign in as another user</a></li>
 	</ul>
 
-	<div class="box box_grey">
+	<div class="card my-4">
-		<h2>Do action</h2>
+		<h2 class="card-header">Do action</h2>
 
-		<form method="post" action="" class="box-body">
+		<form method="post" action="" class="card-body">
 			<input type="hidden" name="csrf_token" value="{{ csrf_token() }}" />
 			<select name="action">
-				<option value="importusers">Create users from mod list</option>
+				<option value="delstuckreleases" selected>Delete stuck releases</option>
-				<option value="importmodlist">Import Krock's mod list</option>
+				<option value="checkreleases">Validate all Zip releases</option>
-				<option value="importscreenshots" selected>Import screenshots from VCS</option>
+				<option value="importmodlist">Import forum topics</option>
-				<option value="importdepends">Import dependencies from downloads</option>
+				<option value="recalcscores">Recalculate package scores</option>
-				<option value="modprovides">Set provides to mod name</option>
+				<option value="checkusers">Check forum users</option>
+				<option value="importscreenshots">Import screenshots from VCS</option>
+				<!-- <option value="importdepends">Import dependencies from downloads</option> -->
+				<!-- <option value="modprovides">Set provides to mod name</option> -->
+				<!-- <option value="vcsrelease">Create VCS releases</option> -->
 			</select>
 			<input type="submit" value="Perform" />
 		</form>
 	</div>
 
-	<div class="box box_grey">
+	<div class="card my-4">
-		<h2>Restore Package</h2>
+		<h2 class="card-header">Restore Package</h2>
 
-		<form method="post" action="" class="box-body">
+		<form method="post" action="" class="card-body">
 			<input type="hidden" name="csrf_token" value="{{ csrf_token() }}" />
 			<input type="hidden" name="action" value="restore" />
 			<select name="package">

app/templates/admin/tags/edit.html

@@ -10,8 +10,8 @@
 
 {% block content %}
 	<p>
-		<a href="{{ url_for('tag_list_page') }}">Back to list</a> |
+		<a href="{{ url_for('admin.tag_list') }}">Back to list</a> |
-		<a href="{{ url_for('createedit_tag_page') }}">New Tag</a>
+		<a href="{{ url_for('admin.create_edit_tag') }}">New Tag</a>
 	</p>
 
 	{% from "macros/forms.html" import render_field, render_submit_field %}

app/templates/admin/tags/list.html

@@ -0,0 +1,16 @@
+{% extends "base.html" %}
+
+{% block title %}
+Tags
+{% endblock %}
+
+{% block content %}
+	<p>
+		<a href="{{ url_for('admin.create_edit_tag') }}">New Tag</a>
+	</p>
+	<ul>
+		{% for t in tags %}
+			<li><a href="{{ url_for('admin.create_edit_tag', name=t.name) }}">{{ t.title }}</a> [{{ t.packages | count }} packages]</li>
+		{% endfor %}
+	</ul>
+{% endblock %}

app/templates/admin/versions/edit.html

@@ -0,0 +1,25 @@
+{% extends "base.html" %}
+
+{% block title %}
+	{% if version %}
+		Edit {{ version.name }}
+	{% else %}
+		New Minetest Version
+	{% endif %}
+{% endblock %}
+
+{% block content %}
+	<p>
+		<a href="{{ url_for('admin.version_list') }}">Back to list</a> |
+		<a href="{{ url_for('admin.create_edit_version') }}">New Version</a>
+	</p>
+
+	{% from "macros/forms.html" import render_field, render_submit_field %}
+	<form method="POST" action="" enctype="multipart/form-data">
+		{{ form.hidden_tag() }}
+
+		{{ render_field(form.name) }}
+		{{ render_field(form.protocol) }}
+		{{ render_submit_field(form.submit) }}
+	</form>
+{% endblock %}