Construction and Testing of the ATLAS Testbed GIIS
(Using MDS version 2.1)
Release Date: 05/21/02
This page shows one how to create a site-level GIIS server and have
that server register with the testbed's GIIS server. The instructions
assume that you have installed Globus Toolkit 2.0 on one or more machines,
have all components other than MDS working correctly, and have permissions
necessary to modify the needed configuration files. Also, it assumed
that the MDS will be started as a user that has permissions to write into
Globus' var directory
If you have not yet installed Globus, I would recommend that you use
a distribution based on the "production" code released in April, 2002.
In particular, this release includes the GRAM reporter as well as speed
enhancements to the LDAP backends.
The bottom of this page contains links to the documentation available
on the Globus website.
Minimal Background:
An MDS system consists of several components; GRIS, GIIS, Information Providers,
LDAP servers and client programs. A GRIS (Grid Resource Information Service)
provides information about a resource to users or agents. The GRIS
collects information about a resource by running Information Providers
that are tuned to the information that is to be gathered and the type and
configuration of the resource. In the way that a GRIS aggregates
information from multiple Information Providers, GIIS can aggregate information
from multiple GRIS. Both GRIS and GIIS are examples of writing custom
backends for the OpenLDAP server. A properly configured LDAP server,
(hereafter referred to by the daemon's name, slapd) will act as both a
GRIS and GIIS. Finally, to read information you need client software
capable of talking to the slapd service. The Globus Toolkit provides
grid-info-search
and grid-info-host-search as client tools. Each command is
a wrapper for the more general ldap_search program provided with
the OpenLDAP software. One should be able to use any client that
follows the LDAP version 3 protocol.
Configuration files:
There are a number of files in $GLOBUS_LOCATION/etc that affect
how the components above behave:
grid-info-resource-ldif.conf
This file details the properties of the Information Providers that
a GRIS will use. Each entry, containing several lines, provides the
path of the information provider, the arguments to use for the information
provider, the dn that will cause the information to run as well as information
that tells GRIS how often the information provider should run.
grid-info-resource-register.conf
This file specifies where a GRIS and GIIS will report their information
in an MDS hierarchy. Each entry specifies a destination for registration
messages, information about the entity (GIIS or GRIS) that is registering
as well as the information concerning the how often a registration message
will be sent.
grid-info-site-giis.conf
This file can be used to set default registrations for a GIIS.
At invocation, a GIIS reads each entry in the file and accepts the each
entry as a registration message. Since registration messages are
sent periodically this file "primes" the GIIS.
grid-info-site-policy.conf
This file configures a GIIS so that only certain registration messages
are accepted. The policydata attribute specified in this file
is an LDAP search filter that operates on the contents of registration
messages. If the registration message causes the filter to be TRUE
the registration is accepted by the GIIS and the registering entity will
be visible under the GIIS.
grid-info-slapd.conf
This file configures how the LDAP server will operate and what services
are being offered by the machine. In particular, the file will determine
whether a GIIS and a GRIS is started when the slapd daemon is run.
A GRIS is available when the file contains an ldif database specification.
A giis database specifies that a GIIS will be started. Also specified
in this file are the schema files that provide rules on the structure and
syntax of allowable information in the system.
Note: The man page for slapd.conf (the basis of grid-info-slapd.conf)
shows how to enable logging by using the loglevel command. This can
be very helpful if the slapd daemon will not start. Note that a value
of -1 for loglevel will turn on full logging. The data is logged
to the local4 facility of syslogd. Once a problem is corrected, you
should remove the loglevel directive and restart the MDS.
grid-info.conf
This file sets various environment variables used
by the system to determine defaults for grid-info-search.
Locations for various components:
$GLOBUS_LOCATION/etc
Contains the files mentioned above as well as the two MDS schema files.
$GLOBUS_LOCATION/etc/openldap/schema
Contains the core.schema file that any LDAP server must recognize
$GLOBUS_LOCATION/libexec
Contains the default information providers, and the slapd executables
$GLOBUS_LOCATION/libexec/openldap/{FLAVOR}/
Contains the GIIS and GRIS backends where {FLAVOR} is a flavor
name specified when the MDS server bundle was built.
$GLOBUS_LOCATION/sbin
Contains the SXXgris and the related grid-info-soft-register scripts.
SXXgris can be used to start and stop the MDS system and will call grid-info-soft-register
as well as starting slapd. Some distributions may have a copy of
SXXgris as /etc/rc.d/init.d/gris
Initial Configuration
If you installed the Information Services Server bundle, you will have
a default setup of a GRIS and a GIIS running on the same machine.
The grid-info-slapd.conf file will contain a database ldif section that
specifies that the GRIS is known by the suffix "Mds-Vo-name = local, O
= Grid" and the GIIS (denoted by the database giis section) is known by
the "Mds-Vo-name = site, O = Grid" suffix. Both GRIS and GIIS run
inside the same slapd server running on port 2135. If you have note
done so, you should make sure that the MDS can be started and will respond
to queries.
Check to see if MDS is running:
$ ps -auxw | egrep slapd
$ ps -auxw | egrep grid-info-soft-register
You should check the first output for processes running $GLOBUS_LOCATION/libexec/slapd.
If MDS is running there are likely several.
The second command checks that the GRIS is registering to the GIIS.
In the default setup there should be two processes that are executing $GLOBUS_LOCATION/sbin/grid-info-soft-register
To start the MDS you should use:
bash$ $GLOBUS_LOCATION/sbin/SXXgris start
Use the ps commands above to verify that the MDS appears to be working.
Next issue the following commands:
bash$ grid-info-search -x
bash$ grid-info-search -x -b"mds-vo-name=site, o=grid"
The first command should provide the contents of the GRIS. Each
line starting with dn: should have a trailing "Mds-Vo-name=local, O=Grid"
The second command should provide the same information except that the
trailing lines should have "MDS-Vo-name=site, O=grid" and there should
be 1 additional entry for a GlobusStub object that lists the GRIS.
If you get this far, your Globus installation is running correctly.
The next step is to create a site-level GIIS that will collect the information
from your GRIS machines.
Creating a site-level GIIS
The idea is to enable and start GRIS on each node that will publish information.
Each GRIS will report to a single GIIS that will aggregate the information
for your site. The simplest strategy is to have a GRIS running on
every node that is running the gatekeeper service. It is assumed
that the gram-reporter information providers, runnning from within GRIS
on the gatekeeper, will provide information about your batch queue.
In order for your site level GIIS to be distinguished at the next higher
level, you will need to name your GIIS. As mentioned above the default
setup is to name a GIIS using "site". Using this name will cause
problems when creating a test-bed level GIIS if more than one installation
uses the name. Instead, it is recommended that your site-level GIIS
name should be based on your domain name. For instance, our site-level
GIIS is named "uta".
Note:
While it appears that the name given to a GIIS is arbitrary the name
must follow a syntax. In particular GIIS names MUST be comprised
of characters taken from the following set (RFC 2252 section 4.1 production
p):
{ A-Z, a-z, 0-9, ", (, ), +, ",", -, ., /, :, ?, " "}
Also, a GRIS should always use the "Mds-Vo-name=local, O=grid" suffix.
This suffix is directly coded within the MDS components.
Step By Step Instructions:
The following instructions assume that you are running a GIIS in
conjunction with a GRIS and that you are modifying the default installation
and that your machines are utilizing NTP for clock synchronization:
For each sub-step the file that needs to be modified is indicated in italics
1) Create a Named GIIS
The instructions for this step are peculiar to the host that will run
both GIIS and GRIS.
A) $GLOBUS_LOCATION/etc/grid-info-slapd.conf
Modify the suffix entry for the GIIS:
suffix "Mds-Vo-name=site,
o=Grid"
to become:
suffix "Mds-Vo-name=<GIIS_NAME>,
o=Grid"
Where <GIIS_NAME> is the name based on your domain.
B) $GLOBUS_LOCATION/etc/grid-info-resource-register.conf (have
the GRIS register to the GIIS)
Modify
dn: Mds-Vo-Op-name=register, Mds-Vo-name=site, o=grid
to become:
dn: Mds-Vo-Op-name=register, Mds-Vo-name=<GIIS_NAME>, o=grid
C) $GLOBUS_LOCATION/etc/grid-info.conf (Modify environment
variables)
Modify
GRID_INFO_ORGANIZATION_DN="Mds-Vo-name=site, o=Grid"
to become:
GRID_INFO_ORGANIZATION_DN="Mds-Vo-name=<GIIS_NAME>, o=Grid"
D) $GLOBUS_LOCATION/etc/grid-info-site-policy.conf
Modify:
policydata: (&(Mds-Service-hn=heppc31)(Mds-Service-port=2135))
to look like:
policydata: (|(Mds-Service-hn=heppc31.uta.edu)
(Mds-Service-hn=heppc6.uta.edu))
Where each there is one (Mds-Service-hn=<host>) for each registering
GRIS
E) Restart GRIS/GIIS on this machine:
bash$ $GLOBUS_LOCATION/sbin/SXXgris stop
bash$ $GLOBUS_LOCATION/sbin/SXXgris start
2) Have other GRIS report to the GIIS
These instructions should be used at the remaining nodes that will
run a GRIS
A) $GLOBUS_LOCATION/etc/grid-info-slapd.conf (remove the
GIIS from this machine)
Remove or comment the lines:
database giis
suffix "Mds-Vo-name=site,
o=Grid"
conf
/opt/globus2/etc/grid-info-site-giis.conf
policyfile /opt/globus2/etc/grid-info-site-policy.conf
anonymousbind yes
access to * by * write
B) $GLOBUS_LOCATION/etc/grid-info-resource-register.conf
(have the GRIS register to the GIIS)
Modify
dn: Mds-Vo-Op-name=register, Mds-Vo-name=site, o=grid
regtype: mdsreg2
reghn: <GRIS_MACHINE>
to become:
dn: Mds-Vo-Op-name=register, Mds-Vo-name=<GIIS_NAME>, o=grid
regtype: mdsreg2
reghn: <GIIS_MACHINE>
where <GRIS_MACHINE> should be the host name of the machine whose files
are being modified.
<GIIS_NAME> is name given to the GIIS in step 1A)
<GIIS_MACHINE> is host name of the machine running the GIIS.
C) $GLOBUS_LOCATION/etc/grid-info.conf (Modify environment variables)
Modify
GRID_INFO_ORGANIZATION_DN="Mds-Vo-name=site, o=Grid"
to become:
GRID_INFO_ORGANIZATION_DN="Mds-Vo-name=<GIIS_NAME>, o=Grid"
D) Restart GRIS on this machine:
bash$ $GLOBUS_LOCATION/sbin/SXXgris stop
bash$ $GLOBUS_LOCATION/sbin/SXXgris start
4) Test your installation:
For each GRIS use:
bash$ grid-info-search -x -h <GRIS_MACHINE> -b"mds-vo-name=local,
o=grid"
You should see about 16 entries listed for a default Globus installation.
To test the GIIS, use:
bash$ grid-info-search -x -h <GIIS_MACHINE> -b"mds-vo-name=<GIIS_NAME>,
o =grid"
5) Register you site level GIIS with the test-bed level GIIS machines:
A) $GLOBUS_LOCATION/etc/grid-info-resource-register.conf (on the
GIIS node)
Add an entry for EACH test bed GIIS that you will register to:
The UTA test-bed GIIS is atlas-giis.uta.edu
dn: Mds-Vo-Op-name=register, Mds-Vo-name=atlas, o=grid
regtype: mdsreg2
reghn: atlas-giis.uta.edu
regport: 2135
regperiod: 600
type: ldap
hn: <GIIS_MACHINE>
port: 2135
rootdn: Mds-Vo-name=<GIIS_NAME>, o=grid
ttl: 1200
timeout: 20
mode: cachedump
cachettl: 30
The BNL test-bed GIIS is giis001.usatlas.bnl.gov
dn: Mds-Vo-Op-name=register, Mds-Vo-name=atlas, o=grid
regtype: mdsreg2
reghn: giis01.usatlas.bnl.gov
regport: 2135
regperiod: 600
type: ldap
hn: <GIIS_MACHINE>
port: 2135
rootdn: Mds-Vo-name=<GIIS_NAME>, o=grid
ttl: 1200
timeout: 20
mode: cachedump
cachettl: 30
where <GIIS_MACHINE> should be the FQDN of your site-level GIIS node.
and <GIIS_NAME> is the name of your GIIS.
B) email the administrators for the test-bed GIIS servers so that
your GIIS registration will be accepted and provide a copy of your grid-info-resource-register.conf
modified in the previous step. You can reach me at mcguigan@hepmail.uta.edu.
Dantong Yu is administering the test-bed giis at BNL and can be reached
at dtyu@bnl.gov
Once you get a reply you should be able to see your site's data included
in the test-bed level GIIS using:
grid-info-search -x -h atlas-giis.uta.edu -b"mds-vo-name=atlas,
o=grid"
to see just your data:
grid-info-search -x -h atlas-giis.uta.edu -b"mds-vo-name=<GIIS_NAME>,
mds-vo-name=atlas, o=grid"
Miscellaneous
Some documentation for the grid-info-search command is here.
Setting up the gram-reporter to report on a condor master is here.
References:
Documentation on creating an hierarchical GIIS structure: