Header
Home lang-en lang-de
Home

Software development
Solutions
References

Consulting
Ericsson PPX (Nortel Passport)
Backup/Restore concepts

download
xml2database

Contact
Contact

xml2database



xml2database_overview.png

xml2database is a flexible, configurable and powerful converter very large amounts (>GByte) of XML data for fast import into databases. A dynamic configuration language allows fast adaption of counters and measurements mapped to database tables. Changes in the naming of XML attributes can be captured within the converter, so that a once-established database tables does not need to be changed if an attribute name has changed on the XML-generating subsystem. This saves money and your time!

You have tested the demo version and would like to purchase a production version of the software or need adjustments, tests at on your reference system or modification configuration files for your individual database tables?

We like to help you:
AD-xml2database.jpeg

Free demo software/documentation download:

Linux Version X86:
OpenSolaris 10 / X86 - distributions Paket: xml2database-OpenSolaris-V0.8.1.tar.gz
OpenSolaris 10 / X86 - demo binary only: xml2database-OpenSolaris-V0.8.1-demo

linux / X86 - distributions Paket: xml2database-linux-V0.8.1.tar.gz
linux / X86 - demo binary only: xml2database-linux-V0.8.1-demo

Handbook: xml2database_user_handbook.pdf

Used technologies:
  • C/C++
  • Qt Bibliothek
  • flex/bison
  • Regular Expression
  • Unix Subprocesse
  • Unix named pipes

Available platforms:
  • Solaris 9 / Sparc
  • OpenSolaris 10 / Sparc
  • Linux / X86

Installation

  • Create a new Unix user anlegen e.g. xml2db
  • Log in the created account
  • Extract the tar file: tar xvf xml2database-linux-x.y.z.tar.gz
  • First test in in bin subdirectory: ./bin/xml2database -v aufrufen
    With option -v the software version will be printed
  • Edit with editor vi the file bin/call_xml2database: Change the environment variable $HOME to the path of the created user: "/home/xml2db"

Test run

  • Change into the root directory of the created user: cd /home/xml2db
  • Inject test data: cp Testdata/* input/
  • Create ssh keys (see handbook "5.4. ssh key generation")
    ssh-keygen -t rsa
    cd .ssh
    cat id_dsa.pub >>authorized_keys2
  • Test the sftp access: sftp xml2db@localhost
    The cron job call_xml2database needs automated sftp access!
  • Start the conversion process: cd bin ; ./call_xml2database
    The converted data will be transfered into the directory fake_ftpserver

Example configuration: 

#
# Software testing
#
INPUTFILE "^TESTDATA.*\.xml$" {                                                           # data source
GENERATES MEASUREMENT {
        MEASUREMENTNAME "DEVELOPMENT_TEST1"                                               # name of the measurement
        GRANULARITY    15MIN                                                              # "15MIN"     -> "Q"
                                                                                          # "24H"        -> "D"
                                                                                          # "EVENT"   -> "E"
        MATCH {
            NEEDS "datarecord"->"shell"
        }
        
        COLUMNS {
            UTC_FILESTAMP                                                "UTCTIME"        # UTC from filename
            REGEXPATTRIBUTE( "Port", "Port ([0-9]*)/[0-9]*/[0-9]*" )     "Shelf"          # extract shelf
            REGEXPATTRIBUTE( "Port", "Port [0-9]*/([0-9]*)/[0-9]*" )     "Slot"           # extract slot
            REGEXPATTRIBUTE( "Port", "Port [0-9]*/[0-9]*/([0-9]*)" )     "Port"           # extract port
            CONVERTATTRIBUTE MS_TO_UTC(    "timeCaptured" )              "UTCTIME1"       # UTC time from milliseconds
            CONVERTATTRIBUTE MS_TO_LOCAL( "timeCaptured" )               "LTIME1"         # localtime from milliseconds
            CONVERTATTRIBUTE STRING_TO_UTC( "StartTime", 
                    "yyyy'-'MM'-'dd'T'hh':'mm':'ss'Z'" )                 "ISO_UTC_TIME"   # UTC time
            CONVERTATTRIBUTE STRING_TO_LOCAL( "StartTime", 
                    "yyyy'-'MM'-'dd'T'hh':'mm':'ss'Z'" )                 "ISO_LOCAL_TIME" # local time
            ATTRIBUTE( "receivedTotalOctets" )                           "RECEIVED"       #
            ATTRIBUTE( "transmittedTotalOctets" )                        "TRANSMITTED"    #
            BOOLATTRIBUTE( "bits"->"subattribute", "active" )            "IS_ACTIVE"      # 1 = if "active" else 0
            ARGATTRIBUTE( "attributeWithArgs", "arg1" )                  "ARGATTR1"
            ARGATTRIBUTE( "attributeWithArgs", "arg3" )                  "ARGATTR3"
        }
    }

    GENERATES MEASUREMENT {
        MEASUREMENTNAME "DEVELOPMENT_TEST2"                                               # Name der Messung
        GRANULARITY    24H                                                                # "15MIN"     -> "Q"
                                                                                          # "24H"        -> "D"
                                                                                          # "EVENT"   -> "E"
        MATCH {
            NEEDS "data"->"shell"->"bits"
        }
        
        COLUMNS {
            UTC_FILESTAMP                                                "UTCTIME"        # UTC from filename
            REGEXPATTRIBUTE( "Port", "Port ([0-9]*)/[0-9]*/[0-9]*" )     "Shelf"          # extract shelf
            REGEXPATTRIBUTE( "Port", "Port [0-9]*/([0-9]*)/[0-9]*" )     "Slot"           # extract slot
            REGEXPATTRIBUTE( "Port", "Port [0-9]*/[0-9]*/([0-9]*)" )     "Port"           # extract port
            CONVERTATTRIBUTE MS_TO_UTC(    "timeCaptured" )              "UTCTIME1"       # UTC time from milliseconds
            CONVERTATTRIBUTE MS_TO_LOCAL( "timeCaptured" )               "LTIME1"         # localtime from milliseconds
            CONVERTATTRIBUTE STRING_TO_UTC ( "ISO8601TIME", 
                    "yyyy'-'MM'-'dd'T'hh':'mm':'ss'Z'" )                 "ISO_UTC_TIME"   # UTC
            CONVERTATTRIBUTE STRING_TO_LOCAL( "ISO8601TIME", 
                    "yyyy'-'MM'-'dd'T'hh':'mm':'ss'Z'" )                 "ISO_LOCAL_TIME" # localtime
            BOOLATTRIBUTE( "bits"->"subattribute", "active" )            "IS_ACTIVE"      # 1 = if "active" else 0
            ATTRIBUTE( "transmittedTotalOctets" )                        "TRANSMITTED"    #
            ATTRIBUTE( "receivedTotalOctets" )                           "RECEIVED"       #
        }
    }
}