opendev/system-config
Code Issues Proposed changes
master
system-config/testinfra/test_lists_opendev_org.py

102 lines
4.5 KiB
Python
Raw Permalink Normal View History

Add a mailman3 list server This should now be a largely functional deployment of mailman 3. There are still some bits that need testing but we'll use followup changes to force failure and hold nodes. This deployment of mailman3 uses upstream docker container images. We currently hack up uids and gids to accomodate that. We also hack up the settings file and bind mount it over the upstream file in order to use host networking. We override the hyperkitty index type to xapian. All list domains are hosted in a single installation and we use native vhosting to handle that. We'll deploy this to a new server and migrate one mailing list domain at a time. This will allow us to start with lists.opendev.org and test things like dmarc settings before expanding to the remaining lists. A migration script is also included, which has seen extensive testing on held nodes for importing copies of the production data sets. Change-Id: Ic9bf5cfaf0b87c100a6ce003a6645010a7b50358
2022-07-27 11:35:41 -07:00
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from util import take_screenshots
testinfra_hosts = ['lists99.opendev.org']
def test_mariadb_listening(host):
mariadb_port = host.socket("tcp://127.0.0.1:3306")
assert mariadb_port.is_listening
def test_mailman3_web_listening(host):
mailman_web = host.socket("tcp://0.0.0.0:8000")
assert mailman_web.is_listening
mailman_uwsgi = host.socket("tcp://0.0.0.0:8080")
assert mailman_uwsgi.is_listening
def test_mailman3_core_listening(host):
mailman_rest = host.socket("tcp://127.0.0.1:8001")
assert mailman_rest.is_listening
mailman_lmtp = host.socket("tcp://127.0.0.1:8024")
assert mailman_lmtp.is_listening
def test_apache2_listening(host):
apache2_http = host.socket("tcp://0.0.0.0:80")
assert apache2_http.is_listening
apache2_https = host.socket("tcp://0.0.0.0:443")
assert apache2_https.is_listening
Add robots.txt to our list servers We've noticed that our uwsgi queues are filling up and a lot of requests are being made to robots.txt which ends up 500/503 erroring. Add a robots.txt file which allows crawling of our lists and archives with a delay value in hopes this will cause bots to cache results and not fill up the queue with repetetive requests. Change-Id: I660d8d43f6b2d96663212d93ec48e67d86e9e761
2024-04-19 13:46:50 -07:00
def test_robots(host):
cmd = host.run('curl --insecure '
'--resolve lists.opendev.org:443:127.0.0.1 '
'https://lists.opendev.org/robots.txt')
assert 'Disallow: /accounts/*' in cmd.stdout
assert 'Allow: /archives/*' in cmd.stdout
Move OpenInfra mailing lists to new domain The OpenInfra Foundation is switching from openinfra.dev to openinfra.org, so move all the mailing lists to prefer the new addresses and URLs while still supporting the previous ones. Also add some redirect testing to exercise the Apache rewrites. Change-Id: Ic10d6519e2a1c4ddab38fdb3119cc20ee62ca741
2024-11-26 19:30:42 +00:00
def test_http_to_https_redirects(host):
cmd = host.run('curl '
'--resolve lists.opendev.org:80:127.0.0.1 '
'http://lists.opendev.org/mailman3/lists/service-discuss.lists.opendev.org/')
assert '301 Moved Permanently' in cmd.stdout
assert 'https://lists.opendev.org/mailman3/lists/service-discuss.lists.opendev.org/' in cmd.stdout
def test_domain_redirects(host):
cmd = host.run('curl '
'--resolve lists.openinfra.dev:80:127.0.0.1 '
'http://lists.openinfra.dev/mailman3/lists/foundation.lists.openinfra.dev/')
assert '301 Moved Permanently' in cmd.stdout
assert 'https://lists.openinfra.org/mailman3/lists/foundation.lists.openinfra.org/' in cmd.stdout
cmd = host.run('curl --insecure '
'--resolve lists.openinfra.dev:443:127.0.0.1 '
'https://lists.openinfra.dev/mailman3/lists/foundation.lists.openinfra.dev/')
assert '301 Moved Permanently' in cmd.stdout
assert 'https://lists.openinfra.org/mailman3/lists/foundation.lists.openinfra.org/' in cmd.stdout
cmd = host.run('curl --insecure '
'--resolve lists.openinfra.dev:443:127.0.0.1 '
'https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/')
assert '301 Moved Permanently' in cmd.stdout
assert 'https://lists.openinfra.org/archives/list/foundation@lists.openinfra.org/' in cmd.stdout
Add a mailman3 list server This should now be a largely functional deployment of mailman 3. There are still some bits that need testing but we'll use followup changes to force failure and hold nodes. This deployment of mailman3 uses upstream docker container images. We currently hack up uids and gids to accomodate that. We also hack up the settings file and bind mount it over the upstream file in order to use host networking. We override the hyperkitty index type to xapian. All list domains are hosted in a single installation and we use native vhosting to handle that. We'll deploy this to a new server and migrate one mailing list domain at a time. This will allow us to start with lists.opendev.org and test things like dmarc settings before expanding to the remaining lists. A migration script is also included, which has seen extensive testing on held nodes for importing copies of the production data sets. Change-Id: Ic9bf5cfaf0b87c100a6ce003a6645010a7b50358
2022-07-27 11:35:41 -07:00
def test_mailman3_screenshots(host):
shots = (
("https://lists.opendev.org:443", None, "mm3-opendev-main.png"),
Fork the maxking/docker-mailman images These images have a number of issues we've identified and worked around. The current iteration of this change is essentially identical to upstream but with a minor tweak to allow the latest mailman version, and adjusts the paths for hyperkitty and postorius URLs to match those in the upstream mailman-web codebase, but doesn't try to address the other items. However, we should consider moving our fixes from ansible into the docker images where possible and upstream those updates. Unfortunately upstream hasn't been super responsive so far hence this fork. For tracking purposes here are the issues/PRs we've already filed upstream: https://github.com/maxking/docker-mailman/pull/552 https://github.com/maxking/docker-mailman/issues/548 https://github.com/maxking/docker-mailman/issues/549 https://github.com/maxking/docker-mailman/issues/550 Change-Id: I3314037d46c2ef2086a06dea0321d9f8cdd35c73
2022-10-03 14:32:23 -07:00
("https://lists.opendev.org:443/archives/",
Add a mailman3 list server This should now be a largely functional deployment of mailman 3. There are still some bits that need testing but we'll use followup changes to force failure and hold nodes. This deployment of mailman3 uses upstream docker container images. We currently hack up uids and gids to accomodate that. We also hack up the settings file and bind mount it over the upstream file in order to use host networking. We override the hyperkitty index type to xapian. All list domains are hosted in a single installation and we use native vhosting to handle that. We'll deploy this to a new server and migrate one mailing list domain at a time. This will allow us to start with lists.opendev.org and test things like dmarc settings before expanding to the remaining lists. A migration script is also included, which has seen extensive testing on held nodes for importing copies of the production data sets. Change-Id: Ic9bf5cfaf0b87c100a6ce003a6645010a7b50358
2022-07-27 11:35:41 -07:00
None, "mm3-opendev-archives.png"),
("https://lists.opendev.org:443"
Fork the maxking/docker-mailman images These images have a number of issues we've identified and worked around. The current iteration of this change is essentially identical to upstream but with a minor tweak to allow the latest mailman version, and adjusts the paths for hyperkitty and postorius URLs to match those in the upstream mailman-web codebase, but doesn't try to address the other items. However, we should consider moving our fixes from ansible into the docker images where possible and upstream those updates. Unfortunately upstream hasn't been super responsive so far hence this fork. For tracking purposes here are the issues/PRs we've already filed upstream: https://github.com/maxking/docker-mailman/pull/552 https://github.com/maxking/docker-mailman/issues/548 https://github.com/maxking/docker-mailman/issues/549 https://github.com/maxking/docker-mailman/issues/550 Change-Id: I3314037d46c2ef2086a06dea0321d9f8cdd35c73
2022-10-03 14:32:23 -07:00
"/archives/list/service-discuss@lists.opendev.org/",
Add a mailman3 list server This should now be a largely functional deployment of mailman 3. There are still some bits that need testing but we'll use followup changes to force failure and hold nodes. This deployment of mailman3 uses upstream docker container images. We currently hack up uids and gids to accomodate that. We also hack up the settings file and bind mount it over the upstream file in order to use host networking. We override the hyperkitty index type to xapian. All list domains are hosted in a single installation and we use native vhosting to handle that. We'll deploy this to a new server and migrate one mailing list domain at a time. This will allow us to start with lists.opendev.org and test things like dmarc settings before expanding to the remaining lists. A migration script is also included, which has seen extensive testing on held nodes for importing copies of the production data sets. Change-Id: Ic9bf5cfaf0b87c100a6ce003a6645010a7b50358
2022-07-27 11:35:41 -07:00
None, "mm3-opendev-list.png"),
("https://lists.opendev.org:443"
Fork the maxking/docker-mailman images These images have a number of issues we've identified and worked around. The current iteration of this change is essentially identical to upstream but with a minor tweak to allow the latest mailman version, and adjusts the paths for hyperkitty and postorius URLs to match those in the upstream mailman-web codebase, but doesn't try to address the other items. However, we should consider moving our fixes from ansible into the docker images where possible and upstream those updates. Unfortunately upstream hasn't been super responsive so far hence this fork. For tracking purposes here are the issues/PRs we've already filed upstream: https://github.com/maxking/docker-mailman/pull/552 https://github.com/maxking/docker-mailman/issues/548 https://github.com/maxking/docker-mailman/issues/549 https://github.com/maxking/docker-mailman/issues/550 Change-Id: I3314037d46c2ef2086a06dea0321d9f8cdd35c73
2022-10-03 14:32:23 -07:00
"/accounts/login/?next=/mailman3/lists/",
Add a mailman3 list server This should now be a largely functional deployment of mailman 3. There are still some bits that need testing but we'll use followup changes to force failure and hold nodes. This deployment of mailman3 uses upstream docker container images. We currently hack up uids and gids to accomodate that. We also hack up the settings file and bind mount it over the upstream file in order to use host networking. We override the hyperkitty index type to xapian. All list domains are hosted in a single installation and we use native vhosting to handle that. We'll deploy this to a new server and migrate one mailing list domain at a time. This will allow us to start with lists.opendev.org and test things like dmarc settings before expanding to the remaining lists. A migration script is also included, which has seen extensive testing on held nodes for importing copies of the production data sets. Change-Id: Ic9bf5cfaf0b87c100a6ce003a6645010a7b50358
2022-07-27 11:35:41 -07:00
None, "mm3-opendev-login.png"),
Move OpenInfra mailing lists to new domain The OpenInfra Foundation is switching from openinfra.dev to openinfra.org, so move all the mailing lists to prefer the new addresses and URLs while still supporting the previous ones. Also add some redirect testing to exercise the Apache rewrites. Change-Id: Ic10d6519e2a1c4ddab38fdb3119cc20ee62ca741
2024-11-26 19:30:42 +00:00
("https://lists.openinfra.org:443", None, "mm3-openinfra-main.png"),
("https://lists.openinfra.org:443/archives/",
None, "mm3-openinfra-archives.png"),
("https://lists.openinfra.org:443"
"/archives/list/foundation@lists.openinfra.org/",
None, "mm3-openinfra-list.png"),
("https://lists.openinfra.org:443"
"/accounts/login/?next=/mailman3/lists/",
None, "mm3-openinfra-login.png"),
Add a mailman3 list server This should now be a largely functional deployment of mailman 3. There are still some bits that need testing but we'll use followup changes to force failure and hold nodes. This deployment of mailman3 uses upstream docker container images. We currently hack up uids and gids to accomodate that. We also hack up the settings file and bind mount it over the upstream file in order to use host networking. We override the hyperkitty index type to xapian. All list domains are hosted in a single installation and we use native vhosting to handle that. We'll deploy this to a new server and migrate one mailing list domain at a time. This will allow us to start with lists.opendev.org and test things like dmarc settings before expanding to the remaining lists. A migration script is also included, which has seen extensive testing on held nodes for importing copies of the production data sets. Change-Id: Ic9bf5cfaf0b87c100a6ce003a6645010a7b50358
2022-07-27 11:35:41 -07:00
("https://lists.openstack.org:443", None, "mm3-openstack-main.png"),
Fork the maxking/docker-mailman images These images have a number of issues we've identified and worked around. The current iteration of this change is essentially identical to upstream but with a minor tweak to allow the latest mailman version, and adjusts the paths for hyperkitty and postorius URLs to match those in the upstream mailman-web codebase, but doesn't try to address the other items. However, we should consider moving our fixes from ansible into the docker images where possible and upstream those updates. Unfortunately upstream hasn't been super responsive so far hence this fork. For tracking purposes here are the issues/PRs we've already filed upstream: https://github.com/maxking/docker-mailman/pull/552 https://github.com/maxking/docker-mailman/issues/548 https://github.com/maxking/docker-mailman/issues/549 https://github.com/maxking/docker-mailman/issues/550 Change-Id: I3314037d46c2ef2086a06dea0321d9f8cdd35c73
2022-10-03 14:32:23 -07:00
("https://lists.openstack.org:443/archives/",
Add a mailman3 list server This should now be a largely functional deployment of mailman 3. There are still some bits that need testing but we'll use followup changes to force failure and hold nodes. This deployment of mailman3 uses upstream docker container images. We currently hack up uids and gids to accomodate that. We also hack up the settings file and bind mount it over the upstream file in order to use host networking. We override the hyperkitty index type to xapian. All list domains are hosted in a single installation and we use native vhosting to handle that. We'll deploy this to a new server and migrate one mailing list domain at a time. This will allow us to start with lists.opendev.org and test things like dmarc settings before expanding to the remaining lists. A migration script is also included, which has seen extensive testing on held nodes for importing copies of the production data sets. Change-Id: Ic9bf5cfaf0b87c100a6ce003a6645010a7b50358
2022-07-27 11:35:41 -07:00
None, "mm3-openstack-archives.png"),
("https://lists.openstack.org:443"
Fork the maxking/docker-mailman images These images have a number of issues we've identified and worked around. The current iteration of this change is essentially identical to upstream but with a minor tweak to allow the latest mailman version, and adjusts the paths for hyperkitty and postorius URLs to match those in the upstream mailman-web codebase, but doesn't try to address the other items. However, we should consider moving our fixes from ansible into the docker images where possible and upstream those updates. Unfortunately upstream hasn't been super responsive so far hence this fork. For tracking purposes here are the issues/PRs we've already filed upstream: https://github.com/maxking/docker-mailman/pull/552 https://github.com/maxking/docker-mailman/issues/548 https://github.com/maxking/docker-mailman/issues/549 https://github.com/maxking/docker-mailman/issues/550 Change-Id: I3314037d46c2ef2086a06dea0321d9f8cdd35c73
2022-10-03 14:32:23 -07:00
"/archives/list/openstack-discuss@lists.openstack.org/",
Add a mailman3 list server This should now be a largely functional deployment of mailman 3. There are still some bits that need testing but we'll use followup changes to force failure and hold nodes. This deployment of mailman3 uses upstream docker container images. We currently hack up uids and gids to accomodate that. We also hack up the settings file and bind mount it over the upstream file in order to use host networking. We override the hyperkitty index type to xapian. All list domains are hosted in a single installation and we use native vhosting to handle that. We'll deploy this to a new server and migrate one mailing list domain at a time. This will allow us to start with lists.opendev.org and test things like dmarc settings before expanding to the remaining lists. A migration script is also included, which has seen extensive testing on held nodes for importing copies of the production data sets. Change-Id: Ic9bf5cfaf0b87c100a6ce003a6645010a7b50358
2022-07-27 11:35:41 -07:00
None, "mm3-openstack-list.png"),
("https://lists.openstack.org:443"
Fork the maxking/docker-mailman images These images have a number of issues we've identified and worked around. The current iteration of this change is essentially identical to upstream but with a minor tweak to allow the latest mailman version, and adjusts the paths for hyperkitty and postorius URLs to match those in the upstream mailman-web codebase, but doesn't try to address the other items. However, we should consider moving our fixes from ansible into the docker images where possible and upstream those updates. Unfortunately upstream hasn't been super responsive so far hence this fork. For tracking purposes here are the issues/PRs we've already filed upstream: https://github.com/maxking/docker-mailman/pull/552 https://github.com/maxking/docker-mailman/issues/548 https://github.com/maxking/docker-mailman/issues/549 https://github.com/maxking/docker-mailman/issues/550 Change-Id: I3314037d46c2ef2086a06dea0321d9f8cdd35c73
2022-10-03 14:32:23 -07:00
"/accounts/login/?next=/mailman3/lists/",
Add a mailman3 list server This should now be a largely functional deployment of mailman 3. There are still some bits that need testing but we'll use followup changes to force failure and hold nodes. This deployment of mailman3 uses upstream docker container images. We currently hack up uids and gids to accomodate that. We also hack up the settings file and bind mount it over the upstream file in order to use host networking. We override the hyperkitty index type to xapian. All list domains are hosted in a single installation and we use native vhosting to handle that. We'll deploy this to a new server and migrate one mailing list domain at a time. This will allow us to start with lists.opendev.org and test things like dmarc settings before expanding to the remaining lists. A migration script is also included, which has seen extensive testing on held nodes for importing copies of the production data sets. Change-Id: Ic9bf5cfaf0b87c100a6ce003a6645010a7b50358
2022-07-27 11:35:41 -07:00
None, "mm3-openstack-login.png"),
)
take_screenshots(host, shots)
Copy Permalink