Additional Fields for the APAC Software Map

The GridAustralia GLUE schema extension supports the fields listed in the GLUE Software Extension twiki page. The plan is to make MDS publish all of the fields in the GridAustralia GLUE schema extension.

Currently, the APAC software map only publishes a subset of the GLUE schema fields. Below are the fields the APAC Software Map support:

• Software package
  • grid-enabled (true/false) - this attribute will tell if the software package on the compute resource has been configured to work on GridAustralia? . Non-grid-enabled software packages will be ignored by the parser which converts the information published by the APAC software map to GLUE schema format.
  • name
  • version
  • module name - name of the module that users need to load to use the software package
  • serial? - can support serial job execution
  • parallel? - can support parallel job execution
  • max number of cpus that can be used on a parallel job

And here are the additional fields we are proposing the APAC Software Map should support:

• Software package - an entity which represents a software package installed on the cluster or a compute node of big SMP boxes
  • list of supported parallel types - openmp, mpi, etc
  • license information - name and description of software license
  • list software executables - the main executables/binaries of the software package. Systems Administrators are only expected to list down all the main (commonly used) executables/binaries the software package. Software packages with no executables can be assumed to be non-grid enabled software packages.
  • list of VOs that can use the package. The name of the VO should preferably be taken from VOMS if the group that is allowed to use the software has an associated VO.

• Software executable
  • name of the executable (for example, mb for MrBayes)
  • parallel - a boolean attribute which will tell the user if the executable can be used on a parallel job
  • supported parallel type - the parallel implementation this software executable supports (e.g. OpenMP, MPI, etc)
  • max number of cpus that can be used on a parallel job

Questions/Issues

1. If a piece of software has e.g. a sequential version, and parallel MPI version and a parallel threads version that can only run on a shared memory machine, are all of these meant to be part of the same software package (but different executables), or different packages?
2. Should the license info be at the level of a package or an executable?

Comments

• The above "Design" and you're mapping to the Sw Map schema match our previous decisions and should meet our current requirements (if not future ones)

• With regards to Q1 it will depend on how we wish to interpret the information on the client side. The information model described above currently allows for all those fields to be set simultaneously, the onus is on the client side to present to the user appropriate options to select between the sub-types. Some business rules would need to be added to prevent the above information model from allowing all those fields to be set simultaneously. Both are valid options. Personally, I am inclined to think of a "package which supports different execution types" rather than consider them as separate packages with basically the same name. So you get one fred (with options = mpi or thread or sequential) rather than fred-mpi, fred-sequential... in the software map (along with having to enter all the same basic information repeatedly as well). Will certainly be more concise a list. This approach seems to suggest a package level license for Q2 as well. -- RobertWoodcock - 18 Sep 2007 - 15:11

• David - I wonder if we do have examples of the quite fine grain restrictions listed above. Do some sites have specific job, cpus, cpus per job restrictions on particular software products ? I would have thought that most of those restrictions are based on commercial licenses, they are either token or job count. We seem to have quite a lot of data to be collected there, the more we ask for, the harder it may be to get it.

• David - I wonder if we really need to treat Packages as being parents of executables. Would it be easier to say that its a just flat executable list, executables may have an attribute that is its package name so we can regroup executables into a package in our view. That would be a lot simpler scale out of the existing system, and simpler might mean easier to get. Only problem I see there is things like Fluent that have several executables, any one of which check out a license from a pool. However, that only affects the total number of licenses available, is that something the grid wants to publish ? Just because a site has 30 Fluent licenses in total thats no reason to assume that 30 are available at any one time. For example, Fluent includes 'fluent' and 'gambit', if we say that 'fluent' has 30 licenses and 'gambit' has 30 licenses but we have a total of only 30, its technically wrong in one aspect. But from the user's point of view, its also right, he could run 30 gambit jobs. Its more likely that he'll miss out becasue some other user is using some of those licenses and we are not attempting to deal with that.

• I think David is right, the list above is too long and not useful in the majority of cases. The more complexity there is, the less likely it is that people are going to enter any information, and the less likely it is that the APAC software map is going to be changed to provide what we want. Total available job seats, user seats or CPUs would only be useful if we can get that information dynamically, e.g. from a license server. If we can, then that's useful info to publish in the MDS. But it's not useful to publish it in the APAC software map, as David points out. So that's extra complexity that I don't think is useful. And if we decide to leave those out, then VOView of a software package is no longer needed - all we really need is a list of what VOs are allowed to use the package (i.e. an ACL), which is useful information to publish in the APAC sofware map. However we need some default value, i.e. the client would need to assume that if the list is empty, it means that anyone can use it, not that no-one can use it. Although in some cases it may just mean that no-one has filled in that field. -- PaulCoddington - 29 Oct 2007

• I think David and I were misunderstanding the issue about package name. Currently, the APAC software map publishes the package name (e.g. BLAST) but NOT the software executable name(s) (e.g. blastp, blastall). So the problem is that this doesn't give the info that we need in the MDS, i.e. what the executable name is - which may vary from site to site. -- PaulCoddington - 29 Oct 2007

• Paul Coddington: I just had a look at the example Gerson added below about what the software map is currently publishing. I note that it specifies that the user has to load modules for both Mr Bayes and LAM. I'm not sure why there is not just one. Is it just for convenience? For grid job submission, do we need to support a list of modules to load, or should one be adequate? But this does flag a point that Gerson raised with me some time ago, about whether we need a separate attribute in the software map about whether the application was accessible via the grid. I thought we didn't need it but now having seen this problem I'm not so sure. Maybe we do need such an attribute, which a sysadmin only ticks if someone has actually tested a grid job submission for the software and gotten it to work, i.e. sorted out all the issues with appropriate modules, execution of parallel jobs, etc. - 19 Nov 2007

• Gerson: Here's an example of what the National Facility publishes about the MrBayes application installed on two of their compute resources

<softwaremap_xml_service version="1.0">
   <site site_code="ANU" site_desc="APAC National Facility @ ANU">
      <host fullname="ac.apac.edu.au">
         <software>
            <package name="MrBayes">
               <version name="3.0B4">
               <shell name="anyshell">
               <queue_resource>-l software=mrbayes</queue_resource>
               <batch_script>module load mrbayes/3.0b4</batch_script>
               <serial>
                  <available>y</available>
               </serial>
               <parallel>
                  <available>y</available>
                  <max_cpus>0</max_cpus>
               </parallel>
            </shell>
            </version>
            </package>
            <package name="MrBayes">
               <version name="3.1">
               <shell name="anyshell">
               <queue_resource>-l software=mybayes</queue_resource>
               <batch_script>module load mrbayes/3.1</batch_script>
               <serial>
                  <available>y</available>
               </serial>
               <parallel>
                  <available>y</available>
                  <max_cpus>0</max_cpus>
               </parallel>
            </shell>
            </version>
            </package>
         </software>
      </host>
      <host fullname="lc.apac.edu.au">
         <software>
            <package name="MrBayes">
               <version name="3.0">
               <shell name="None">
               <queue_resource>None</queue_resource>
               <batch_script>None</batch_script>
               <serial>
                  <available>n</available>
               </serial>
               <parallel>
                  <available>n</available>
                  <max_cpus>0</max_cpus>
               </parallel>
            </shell>
            </version>
            </package>
            <package name="MrBayes">
               <version name="3.1">
               <shell name="anyshell">
               <queue_resource>-l software=mrbayes</queue_resource>
               <batch_script>module load mrbayes/3.1
module load lam/7.1.1Intel81</batch_script>
               <serial>
                  <available>y</available>
               </serial>
               <parallel>
                  <available>y</available>
                  <max_cpus>0</max_cpus>
               </parallel>
            </shell>
            </version>
            </package>
         </software>
      </host>
   </site>
</softwaremap_xml_service>

<softwaremap_xml_service version="1.0">
     <site site_code="ANU" site_desc="APAC National Facility @ ANU">
          <host fullname="ac.apac.edu.au">
                <software>
                     <package name="MrBayes">
                           <version name="3.0B4">
                           <module>mrbayes</module>
                           <supported_parallel_types>
                                  <parallel_type>mpi</parallel_type>
                           </supported_parallel_types>
                           <license_info>opensource</license_info>
                           <access_control_list>
                                  <vo>/APACGrid/*</vo>
                           </access_control_list>
                           <executables>
                                   <executable name=mb-seq>
                                   <supports_parallel>no</supports_parallel>
                                   <supported_parallel_types/>
                                   <max_cpus>1</max_cpus>
                           </executable>
                           <executable name=mb>
                                   <supports_parallel>yes</supports_parallel>
                                   <supported_parallel_types>
                                   <parallel_type>mpi</parallel_type>
                                   </supported_parallel_types>
                                   <max_cpus>8</max_cpus>
                           </executable>
                         </executables>
                     </package>
                </software>
           </host>
     </site>
</softwaremap_xml_service>

-- GersonGalang - 02 Dec 2007