Welcome • Services • Getting Started • Support and Tools • Documentation  
 
 

CalNetAD Directory Integration: Technical Design

Updated: 2004-11-05

Summary

We have implemented a phased integration of the data in our campus CalNet Directory (which is based on a Sun LDAPv3 directory server) with our campus Active Directory (CalNetAD). We describe the design and coding in this document. Basic user account information is synchronized between the two directories as detailed in the table of data elements below. Our implementation is based on the IBM Directory Integrator (IDI) engine and custom ADSI scripting using primarily JScript code.

The synchronization process constantly monitors the changelog for the CalNet Directory and sends events of interest as messages to a FioranoMQ JMS Topic maintained by the CalNet Messaging system. This means that any CalNet account change, including CalNet ID changes, will be automatically synchronized and available to the CalNetAD within about a minute. To validate any CalNetAD changes that may have occurred to user accounts via normal administrative activites, a nightly comparison of user data from the two directories is made. This procedure creates a local repository for the CalNetAD user data using the IDI "Delta mechanism" feature. Any differences found in comparison with the authoritative data in the CalNet directory are also sent into the synchronization process to be corrected in the CalNetAD.

Data elements

The following data elements are being synchronized between the CalNet Directory and the CalNetAD:

User attribute: GUI label: Unique within: Initial values (CNID=CalNetID):
altSecurityIdentities Kerberos Principal Name Forest Kerberos:CNID@BERKELEY.EDU
cn Full name OU CNID (when first created)
displayName Display name [not unique] "displayname" from CalNet directory (CNID for students)
mail E-mail [not unique] "mail" from CalNet directory
sAMAccountName User logon name (pre-Win2K) Domain CNID
userPrincipalName User logon name Forest CNID@BERKELEY.EDU
uid [none] Forest "uid" from CalNet directory

 

CNDirPub31: data from the Sun LDAP3 directory

The integration process monitors the Sun directory's changelog for events of interest. The events are processed using an IDI AssemblyLine (AL), CNDirPub31AL. The AL consists of five connectors: CNinput, CNlookup, FMQoutput, FSoutput, and CNinit. The latter connector is a passive-mode connector invoked by custom code in the prolog of CNDirPub31AL to initialize the "nsChangenumber" parameter for the CNinput connector to the changelog. The CNinput and CNlookup connectors both retrieve user data from the directory which is passed through the AL to the output connectors. One output connector, FMQoutput, passes events as messages to a JMS Topic on a FioranoMQ (FMQ) messaging server. A second connector, FSoutput, records all events to a local file in the "LDIF" data format. See the Appendix for a detailed description of CNDirPub31AL and its connectors. This system of directory data events accessible via JMS messages is termed CNDirPub31, and is designed to support multiple data consumers of which the CalNetAD service is the first working example.

CNDirPub31AL

CNADSub31: using an ADSI-based script connector to the CalNetAD

To process the update messages from the CNDirPub31 system, an IDI JMS connector, input passes the data received to an AL, CNADSub31AL, which uses a custom script connector, ADSIScriptConnector, to make add, delete, and modify transactions into the CalNetAD via the add_user, del_user, and mod_user IDI connectors. The script connector is implemented as a set of JScript functions which use ADSI interfaces to query and update the CalNetAD data to comply with our requirements for data synchronization and security and privacy policies. See the Appendix for a detailed description of CNADSub31AL and its custom script connector, ADSIScriptConnector. The system of IDI components which receive and process the directory data to keep the CalNetAD synchronized with the campus CalNet Directory is called the CNADSub31.

CNADSub31AL

ADVal31: checking data integrity

Because data within the CalNetAD can be changed by other processes outside of the publisher/subscriber cycle, a method to validate against the data source of record was required. To begin the validation process, a complete LDIF dump of selected attributes for all user objects in the CalNetAD is created each night using the LDIFDE AD utility. This file is securely transferred to the ADVal31 site. Then, an IDI AL, ADVal31aAL, parses the LDIF file using a Delta B-tree database to record all entries. The epilog of this AL starts a second AL, ADVal31bAL, which then parses a nightly LDIF dump of the CalNet Directory against the same Delta B-tree database to generate any differences as add, delete, or modify operations saved as a local output LDIF file. Finally, the epilog of this second AL starts a third AL, ADVal31cAL, which reads as input the LDIF file of changes and sends each transaction as a message to a JMS Topic on a FioranoMQ (FMQ) messaging server. See the Appendix for a detailed description of ADVal31AL and its connectors. This system of CalNetAD data validation using JMS messages is termed ADVal31, and is designed to coexist with multiple other related data publishers to the same JMS Topic, of which the CNDirPub31 system described above is one such publisher.

ADVal3aAL

ADVal3bAL

ADVal3bAL

 

Policies

The following policies relating to data conversions and security and privacy of data elements have been implemented:

  • All employee (faculty and staff) and affiliate user accounts are by default created within the OU=Users,OU=FSA container
  • All student accounts are created by default within the OU=Users,OU=Students container using the CalNetID as the "displayName" attribute
  • The "cn" attribute is not modified automatically following the initial creation unless the account to be changed still resides within the FSA or any Students OU
  • Student accounts are created to have access control entries (ACEs) which prohibit access to the "uid" attribute for general users
  • Until a mechanism is developed to secure this attribute for student accounts in compliance with FERPA requirements, the "mail" attribute is synchronized only for faculty, staff, and affiliates.
  • If a modification involves moving an account into a Students OU, the ACEs mentioned above are added; if an account is moved out of a Students OU, the ACEs are removed

Appendix

Details for the IDI AssemblyLines, EventHandlers, connectors and the Java Service Wrapper as currently implemented are documented in this Appendix. XML-based IDI export files for the IDI code is available under the following links:

All custom code for the CNDirPub31 and ADVal31 systems use the native IDI Rhino-based Javascript interpreter. The CNADSub31 system uses the IDI support for WSH languages and uses the ECMAScript-based JScript language to facilitate access to ADSI COM-based objects.

CNDirPub31AL

This AL uses an LDAP Changelog connector, CNinput, in Iterator mode to read the CalNet Directory changelog making the following input mappings between event attributes (on the left side) and work entry attributes:

1. changes --> changes
2. changetype --> changetype
3. targetdn --> targetdn

For each event of interest, a lookup is performed using the CNlookup LDAP connector and using the "targetdn" attribute from the changelog as the "Link Criteria". Connector attributes (on the left) are mapped to work attributes as follows:

1. displayname --> displayname
2. [custom code] --> is_student
3. mail --> mail
4. [custom code] --> primaryCalNetID
5. ucbaffiliations --> ucbaffiliations
6. ucbkerberosprincipalstring --> ucbkerberosprincipalstring

Custom code in the Before Lookup and After Lookup IDI AL hooks are used to test incoming entries for validity and to set various flags and values used in the custom code during the attribute mappings.

Two output connectors, FMQoutput and FSoutput, in AddOnly mode send the work objects into both a JMS Topic as a message and into an LDIF data file, CNDirPub31Output.txt, with the following attribute mappings between the work object (on the left side) and the output records:

1. changetype --> action
2. [custom code] --> displayname
3. is_student --> is_student
4. mail --> mail
5. primaryCalNetID --> primaryCalNetID
6. ucbaffiliations --> ucbaffiliations
7. ucbkerberosprincipalstring --> ucbkerberosprincipalstring
8. [custom code --> uid

In addition, for FMQoutput, a JMS custom header, "Source" is defined to indicate the data source. Custom code in the Before Add AL hook ensures that only events of interest are added to the output data connections.

The string values for action are "add", "modify", or "delete", based on the "changetype" value returned by the CNinput connector to the changelog.

CNADSub31AL

This AL receives messages from its JMS connector, input and maps the incoming entry attributes (on the left side) into the work entry attributes as follows:

1. [custom code] --> UPN
2. action --> action
3. [custom code] --> altsecid
4. [custom code] --> cn_default
5. [custom code] --> displayname
6. is_student --> is_student
7. [custom code] --> key
8. mail --> mail
9. [custom code] --> ou_default
10. [custom code] --> samaccount
11. uid --> uid

The work entry then is processed by one of three connectors, add_user, del_user, or mod_user, to perform the transaction within the CalNetAD. These three connectors are all based on the custom Script Connector, ADSIScriptConnector, and operate in AddOnly, Delete, and Update modes, respectively. The add_user and mod_user connectors both make the following mappings of the work object attributes (on the left side) to the Connection object attributes sent to the custom script connector for processing:

1. UPN --> UPN
2. altsecid --> altsecid
3. cn_default --> cn_default
4. displayname --> displayname
5. is_student --> is_student
6. key --> key
7. mail --> mail
8. ou_default --> ou_default
9. samaccount --> samaccount
10. uid --> uid

To control which action is performed, custom code within the Before Execute IDI AL hook retrieves the value of the work object's action attribute and ignores the entry if it doesn't match the appropriate operation for the connector. For example:

// Ignores entries not for MODIFY operation

if (work.getString("action") == ACTION_MOD)
{
  oAttr = work.getAttribute(ALTSECID_ATTR);
  sBuffer = work.getString(UPN);

  if ((oAttr == null) || (sBuffer == null))
  {
    task.logmsg("throw rseEntrySkipped");
    system.skipEntry();
  }
  else
  {
    task.logmsg("UPDATING:");
    task.dumpEntry(work);
  }
}
else
{
    task.logmsg("throw rseEntryIgnore");
    system.ignoreEntry();
}

ADSIScriptConnector

This JScript custom Script Connector implements the required functions for any IDI Script Connector:

  • selectEntries() [stub only]
  • getNextEntry() [stub only]
  • putEntry()
  • findEntry()
  • modEntry()
  • deleteEntry()

In addition, several helper functions are implemented to assist in utility, security, privacy, or error control operations:

  • pack(Ary) - converts JScript array to Scripting.Dictionary object
  • checkError(sOperation, sErr, iCount, sAction) - general ADSI and COM error handling
  • randPass(sPWChars, iSize) - creates random password string
  • searchError(sErr, iCnt, sAct) - search error handling
  • modifyACL(oUsr, sAct) - remove or add ACE for AD user object ACL
  • restoreACE(oUsr, sAct) - restore ACE to AD user object ACL

These latter two functions were implemented to allow privacy policies to be reflected in the access control lists (ACLs) for the uid property of AD user objects.

To access the AD for searching, the ADODB.Connection object is used with the ADsDSOObject ADSI provider interface. Standard ADSI interfaces are used for adding, modifying, and deleting user objects. A special End-of-File (EOF) entry is processed by the putEntry() function to signal the end of data input to the parent IDI AL process in the case of a batch set of transactions as occurs with the ADVal31 process.

Most of the policies relating to security and privacy are implemented as part of the putEntry() function (setting initial random passwords and initial ACEs), or within the modEntry() function (adding or removing ACEs and setting the value for the cn attribute).

ADVal31AL

To control which output file ADVal31bAL selects for a given work object, the operation code value is determined by custom code within the Before Add IDI AL hook for the save_* connectors. The two output files are later merged within the AL epilog into a single data file, ADVal31Save.ldif, which contains all of the "delete" transactions listed first. This technique was chosen to avoid errors encountered when the corrected records for certain types of data errors in the source CalNet Directory were propagated to the CalNetAD in the default order generated by the IDI Delta process in which deletes are placed last in the sequence of transactions.

As for CNDirPub31, the output JMS connector in ADVal31cAL, send, adds a custom header, Source, to each message to allow subscribers to filter messages based on the data source. The attribute mappings are thus quite similar to those for the CNDirPub31 system.

Java Service Wrapper

To support running the IDI ALs as either a Windows service or as a UNIX daemon process we use the Java Service Wrapper (JSW) on our Windows or Solaris application servers. JSW uses a text-based configuration file to control the launching and monitoring of the JVM and to run the IDI application classes with the proper environment paths and settings.

Under Windows, this same configuration file is used by JSW for installing and removing the custom IDI service from the Windows system. Most of the parameters in the JSW configuration file are derived from the batch files that IBM includes for the manual startup of the IDI application. We customized the JSW configuration files and the JSW template batch or shell script files to start the IDI system from the home IDI directory and to include support for logging to the Windows event logs when running on that platform. An example version of the JSW configuration file for CNADSub31 is given below:

wrapper-cnadsub31.conf 

#********************************************************************
# Wrapper Properties
#********************************************************************
# modified 2004-10-11
# Karl R. Grose, IST-WSS

# Environment variables
set.FMQ_DIR=E:\Fiorano\FioranoMQ7.2
set.IDI_DIR=E:\IBM\IDI471
set.JAVA_HOME=C:\Program Files\JavaSoft\JRE\1.3.1_08

# Java Application
wrapper.java.command=%JAVA_HOME%/bin/java

# Java Main class.  This class must implement the WrapperListener interface
#  or guarantee that the WrapperManager class is initialized.  Helper
#  classes are provided to do this for you.  See the Integration section
#  of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp

# Wrapper working directory
wrapper.working.dir=../

# Java Classpath (include wrapper.jar)  Add class path elements as
#  needed starting from 1
wrapper.java.classpath.1=libs/wrapper.jar
wrapper.java.classpath.2=%FMQ_DIR%/lib/common.zip
wrapper.java.classpath.3=%FMQ_DIR%/lib/jndi.jar
wrapper.java.classpath.4=%FMQ_DIR%/lib/fmprtl.zip
wrapper.java.classpath.5=%FMQ_DIR%/lib/httpclient.zip
wrapper.java.classpath.6=%FMQ_DIR%/lib/jcert.jar
wrapper.java.classpath.7=%FMQ_DIR%/lib/jnet.jar
wrapper.java.classpath.8=%FMQ_DIR%/lib/jsse.jar
wrapper.java.classpath.9=%FMQ_DIR%/lib/jndi
wrapper.java.classpath.10=jars/activation.jar
wrapper.java.classpath.11=jars/comm.jar
wrapper.java.classpath.12=jars/imap.jar
wrapper.java.classpath.13=jars/jaas.jar
wrapper.java.classpath.14=jars/ldap.jar
wrapper.java.classpath.15=jars/ldap12.jar
wrapper.java.classpath.16=jars/mail.jar
wrapper.java.classpath.17=jars/mailapi.jar
wrapper.java.classpath.18=jars/ncsow.jar
wrapper.java.classpath.19=jars/pop3.jar
wrapper.java.classpath.20=jars/providerutil.jar
wrapper.java.classpath.21=jars/smtp.jar
wrapper.java.classpath.22=jars/xalan.jar
wrapper.java.classpath.23=jars/xerces.jar
wrapper.java.classpath.24=miloader.jar
wrapper.java.classpath.25=jars/ldapbp.jar
wrapper.java.classpath.26=jars/sunjce_provider.jar
wrapper.java.classpath.27=jars/jce1_2_2.jar
wrapper.java.classpath.28=jars/local_policy.jar
wrapper.java.classpath.29=jars/US_export_policy.jar
wrapper.java.classpath.30=jars/ibmjlog.jar
wrapper.java.classpath.31=%FMQ_DIR%/lib/csp.zip
wrapper.java.classpath.32=%FMQ_DIR%/lib/jta.jar
wrapper.java.classpath.33=%FMQ_DIR%/lib/cryptix.jar

# Java Library Path (location of Wrapper.DLL or libwrapper.so)
wrapper.java.library.path.1=libs

# Java Additional Parameters
wrapper.java.additional.1=-Duser.dir=%IDI_DIR%

# Initial Java Heap Size (in MB)
#wrapper.java.initmemory=3

# Maximum Java Heap Size (in MB)
#wrapper.java.maxmemory=64

# Application parameters.  Add parameters as needed starting from 1
wrapper.app.parameter.1=com.metamerge.loader.miloader
wrapper.app.parameter.2=com.architech.RS
wrapper.app.parameter.3=-cconf/CNADSub31/CNADSub31.cfg
wrapper.app.parameter.4=-rCNADSub31AL
wrapper.app.parameter.5=-lconf/CNADSub31/logs/cnadsub31.log

#********************************************************************
# Wrapper Logging Properties
#********************************************************************
# Format of output for the console.  (See docs for formats)
wrapper.console.format=PM

# Log Level for console output.  (See docs for log levels)
wrapper.console.loglevel=INFO

# Log file to use for wrapper output logging.
wrapper.logfile=logs/wrapper.log

# Format of output for the log file.  (See docs for formats)
wrapper.logfile.format=LPTM

# Log Level for log file output.  (See docs for log levels)
wrapper.logfile.loglevel=INFO

# Maximum size that the log file will be allowed to grow to before
#  the log is rolled. Size is specified in bytes.  The default value
#  of 0, disables log rolling.  May abbreviate with the 'k' (kb) or
#  'm' (mb) suffix.  For example: 10m = 10 megabytes.
wrapper.logfile.maxsize=0

# Maximum number of rolled log files which will be allowed before old
#  files are deleted.  The default value of 0 implies no limit.
wrapper.logfile.maxfiles=0

# Log Level for sys/event log output.  (See docs for log levels)
wrapper.syslog.loglevel=INFO

#********************************************************************
# Wrapper NT Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
#  using this configuration file has been installed as a service.
#  Please uninstall the service before modifying this section.  The
#  service can then be reinstalled.

# Name of the service
wrapper.ntservice.name=IDI_CNADSub31

# Display name of the service
wrapper.ntservice.displayname=IBM DI CNADSub31 Service

# Description of the service
wrapper.ntservice.description=IBM Directory Integrator CNADSub31 Service

# Service dependencies.  Add dependencies as needed starting from 1
wrapper.ntservice.dependency.1=

# Mode in which the service is installed.  AUTO_START or DEMAND_START
wrapper.ntservice.starttype=AUTO_START

# Allow the service to interact with the desktop.
wrapper.ntservice.interactive=false

# The account to use when running the service.
wrapper.ntservice.account=xxxxx

# The password of the account to use when running the service.
wrapper.ntservice.password=xxxxx

FioranoMQ and IDI using SSL

To configure the IDI JMS connectors for using the FioranoMQ system over SSL, we took the following steps:

  1. Assume an FMQ 6.0 or 7.2 installation in /usr/local/FioranoMQ and an IDI 4.7.1 installation in /usr/local/IDI.
  2. Create a jndi.properties file, for example, in /usr/local/FioranoMQ/lib/jndi, containing values for the InitalContext object:
    java.naming.security.protocol=SUN_SSL SSL_SECURITY_MANAGER=JSSESecurityManager [FMQ 6.0] SecurityManager=JSSESecurityManager [FMQ 7.2]
    and add its location to the CLASSPATH (see below).
  3. Modify the miadmin.bat and miserver.bat files (or the Java Service Wrapper configuration file) to include the FMQ jar and zip files and the location of the jndi.properties file in the CLASSPATH:
    FMQ_DIR=/usr/local/FioranoMQ JSSE_CLASSPATH=$FMQ_DIR/lib/jcert.jar:$FMQ_DIR/lib/jnet.jar:$FMQ_DIR/lib/jsse.jar

    myclasspath=$FMQ_DIR/lib/common.zip:$FMQ_DIR/lib/jndi.jar:$FMQ_DIR/lib/fmprtl.zip: $FMQ_DIR/lib/httpclient.zip:$JSSE_CLASSPATH:$FMQ_DIR/lib/jndi:[rest of IDI jars]:
  4. Define Java system properties in the IDI global.properties file to locate the FMQ certificates keystore and related items:
    javax.net.ssl.trustStore=/usr/local/FioranoMQ/bin/jssecacerts FMQ_DIR=/usr/local/FioranoMQ
 
Contact Us