SourceForge Installation Guide - Chapitre 6
Using the Backend

6.1 Introduction

Sourceforge dumps a lot of information from its database into text files which will be parsed later by other scripts. This is the main role of the backend. It is composed of several Perl scripts that you can find in SF_ROOT/utils/underworld-* and some text files (in SF_ROOT/backend/zones) which are the head of the dumped file.

As we said, the current SF implementation procedes in two steps. On our one-machine Sourceforge, we will modify the scripts so we can do the whole thing in only one step.

6.2 Configuring include.pl

This files resides in SF_ROOT/utils and will be used by every backend scripts.You have to configure some variables, here is what we need.

	$db_include =   "SF_ROOT/etc/local.inc";    # Local Include file for database us
	$tar_dir    =   "/tmp";         # Place to put deleted user's accounts
	$uid_add    =   "1000";        # How much to add to the database uid to get the
	$gid_add    =   "2000";         # How much to add to the database gid to get the
	$homedir_prefix = "SF_ROOT/home/users/";     # What prefix to add to the user's homed
	$grpdir_prefix  =   "SF_ROOT/home/groups/";    # What prefix to add to the user's homed
	$file_dir   =   "SF_ROOT/dumps/";   # Where should we stick files we're work
	$dummy_uid      =       "65534";    #  A user that nobody can be (usually nobody)
	$dummy_name     =   "nobody";
	$dummy_group_id     =       "65534"; # A group that has no members (usually nogroup)
	$dummy_group_name       =   "nogroup";
	$date           =       int(time()/3600/24);    # Get the number of days since 1
	$cvsroot   = "SF_ROOT/cvs"; 
	$cvslogs_dir = "SF_ROOT/cvslogs";
	$boalogs_dir = "SF_ROOT/boalogs";
	$download_dir = "SF_ROOT/download"; 
	$domainname = "DOMAIN_NAME";
	$IP = "10.0.0.1"; # The machine IP

6.3 Configuring DNS backend

6.3.1 Working on SourceForge Files

The perl script is SF_ROOT/utils/underworld-dummy/dns_conf.pl. Make it executable.

In order to use the backend, you have to write a head file and copy it to SF_ROOT/utils/zones/dns.zone. You should maybe just edit the example provided with the patch and modify it to fit your own needs. Note that the second line MUST BE the serial number line.

6.3.2 Configuring BIND

Add this to your named.conf

	zone "DOMAIN_NAME" {
 		type master;
    file "/etc/bind/db.DOMAIN_NAME";
	};

Now you link /etc/bind/db.DOMAIN to SF_ROOT/dumps/dns.dump. and create a cron job :

	0 * * * * cd SF_ROOT/utils/underworld-dummy/ && ./dns_conf.pl && ndc -q	reload

6.4 User and group backend

This part has been modified to use the MySQL NSS library (this is discussed in the next chapter).

Add a cronjob for the backend script :

	*/10 * * * * cd SF_ROOT/utils && ./new_parse.pl >/dev/null

Don't forget to customize SF_ROOT/utils/default_page.php ! (for example remove all occurrences of sourceforge.net and replace them by DOMAIN_NAME).

6.5 CVS stats Backend

Make $SF_ROOT/utils/cvs1/cvs_history_parse.pl executable and add a cron job for it

	30 0 * * * cd SF_ROOT/utils/cvs1 && ./cvs_history_parse.pl >/dev/null

6.6 SSH Keys backend

Now add a cron job for the backend script

	*/5 * * * * cd SF_ROOT/utils/ && ./ssh_create.pl > /dev/null

6.7 Download Statistics

First you need to install the Boa server (see http//www.boa.org) and cronolog (see http://www.ford-mason.co.uk/resources/cronolog/).

We have to configure Boa. You just need to customize SF_ROOT/etc/boa.conf and copy it to /etc/[boa].Then add a cron job for the stats backend

	0 1 * * * cd SF_ROOT/utils/download && ./stats_logparse.sh >/dev/null

6.8 WWW backend

We need a script to create the Virtual Host entries. This is done by SF_ROOT/utils/apache.pl. Copy your httpd.conf to SF_ROOT/utils/zones/httpd.conf.zone.

Now add a cron job for the WWW backend

	0 * * * * cd SF_ROOT/utils/ && ./apache.pl > /dev/null & & apachectl restart >/dev/null

6.9 Mailing Lists

6.9.1 Backend configuration

The Sourceforge mailing lists system uses GNU Mailman. So you will have to install it. Then we need to customize SF_ROOT/utils/mailing_lists_create.pl. Indeed it is very dependent of your Linux distribution. Note that I've removed the Geocrawler stuff since only the Sourceforge hosts are allowed to add lists there.

Don't forget to add a cron job for this script

	*/10 * * * * cd SF_ROOT/utils && ./mailing_lists_create.pl > /dev/null

6.9.2 Exim configuration

You need to create aliases for all mailing lists. As always, we are going to use the MySQL features of Exim. First add lists.DOMAIN_NAME to your local domains. Then you have to add three directors (after forward_for_sourceforge) : (a sample is included in SF_ROOT/misc/exim.directors)

forward_for_lists:
  domains = lists.DOMAIN_NAME
  driver = aliasfile
  file_transport = address_file
  pipe_transport = address_pipe
  query = "select CONCAT('|/usr/lib/mailman/mail/wrapper post ',list_name) 
       from mail_group_list where list_name = '$local_part'"
  search_type = mysql 
	user=root
	group=root

forward_for_lists_admin:
  domains = lists.DOMAIN_NAME
  driver = aliasfile
  file_transport = address_file
  pipe_transport = address_pipe
  query = "select CONCAT('|/usr/lib/mailman/mail/wrapper mailowner ',list_name) 
    from mail_group_list where list_name = 
    LEFT('$local_part',LENGTH('$local_part')-6) and 
    RIGHT('$local_part',6) = '-admin'"
  search_type = mysql 
	user=root
	group=root

forward_for_lists_request:
  domains = lists.DOMAIN_NAME
  driver = aliasfile
  file_transport = address_file
  pipe_transport = address_pipe
  query = "select CONCAT('|/usr/lib/mailman/mail/wrapper mailcmd ',list_name) 
    from mail_group_list where list_name = 
    LEFT('$local_part',LENGTH('$local_part')-8) and 
    RIGHT('$local_part',8) = '-request'"
  search_type = mysql 
	user=root
	group=root

As for the mail configuration, you need to give the right to select list_name in mail_group_list to the same user.

6.9.3 Mailman configuration

We suggest that you edit mm_cfg.py (in /etc/ or /etc/mailman/) and set DEFAULT_HOST_NAME to your DOMAIN_NAME and DEFAULT_URL to lists.DOMAIN_NAME/mailman.

6.10 Misc

Add as cron jobs

	0 * * * * cd SF_ROOT/utils/underworld-root && ./db_trove_treesums.pl
	0 2 * * * cd SF_ROOT/utils/underworld-root && ./stats_nightly.sh >/dev/null
	0 2 * * * cd SF_ROOT/utils/underworld-root && ./db_jobs_close.pl
	@daily cd SF_ROOT/utils/underworld-root/ && (./db_project_metric.pl ;
														./db_project_cleanup.pl)
	@daily cd SF_ROOT/utils/underworld-root && db_project_weekly_metric.pl >/dev/null
	@daily cd SF_ROOT/utils/underworld-root && db_rating_stats.pl
	@daily cd SF_ROOT/utils/underworld-root && db_top_groups_calc.pl>/dev/null
	@daily cd SF_ROOT/utils/underworld-root && db_site_stats.pl