True it's been a mystery lurking on our Data Integrator server for some time.

But more than perplexing, it was really annoying.  As part of the on-going to process to better understand, control and hopefully make use of the web based components installed on a Data Integrator server, I bumped into a vexing, albeit, innocuous security related issue.  As mentioned in the previous post, the Data Integrator installation utilizes Tomcat for web-based admin and management features.  And in the course of exploring the web serving components of that environment, occasionally a login dialog boxed would pop up requesting a user ID and password to proceed.  At first, I didn't even pay attention to exactly what the URL (URI) was that would cause this "oddity". 

Everything important related to the operation of Data Integrator has been working fine and accessing its web admin via the appropriate URL did not cause the login pop-up to appear so there's been no great urgency to find the cause of the mysterious dialog box.  In addition, the DI installation does not use the standard HTTP port of 80 to bring up the main admin page, instead using 28080.  While experimenting with several different Data Integrator URLs, all including the custom port reference, to access different DI admin features, I tried just the server name without any port reference and nothing came up.  Just the standard page not found message.

Knowing that DI was being served via Tomcat as well as the fact that Tomcat use port 8080 as a default , I tried a URL that included the server name and that port such as http://server:8080.  And viola, the mysterious login dialog box showed itself.  Gotcha!  Now we know what ever is throwing the login request is listening on 8080 and it definitely doesn't look like its Tomcat.

But what have I really found out?  What the heck is throwing up this login box?  Data Integrator has its own log in page and it works fine sans the dialog box when accessed through proper URL including the custom port assignment.  Time to get a clue.  And there was one right on the login box: "XDB" was displayed above the text entry fields in the dialog.  But again, now what?  Hmmm... XDB...  XDB...  What's that coming from?  And it looks familiar too.  Wait a minute, I've seen XDB.  It's schema in an Oracle 9iR2 database.

Not only is it a schema in Oracle 9 and higher, it's part of of entire XML feature set now standard with Oracle known as Oracle XDB.  Oracle XDB provides for built-in XML capabilities directly integrated within the Oracle database environment.  After researching I decided that perhaps the dialog box wants the credentials associated with XDB schema (I realize when properly configured other login credentials would likely work as well, but no part of XDB in our installation has been specifically configured at this point other than what goes in as default with database installation).  So I did an ALTER USER on XDB  to unlock the account and reset the password to give it try.

Thinking I've got it figured out and expecting a pleasant outcome I try it.  No dice.  What? This can't be right.  Must be a mistake; I'll try again.  Nope.  The dialog just keeps coming back with every attempt eventually resulting in an "unauthorized" page.  Son of gun...  More reading and a lot of Google uncovers that this is not a particularly unique situation, but the answers as to the cause and the solution are elusive.

Our Data Integrator, like many others, uses Oracle as its back-end repository (not familiar myself if other DBMS can be used for the repository but I suspect so).  And Oracle XDB is part of the Oracle installation by default.  But to further mask the cause in our particular situation, we have another separate Oracle instance running on the same box along with the Oracle instance used by Data Integrator.  It was in this other instance in which I originally changed the XDB login and it turns out this was not the instance listening on 8080, thus no access.

When I opened the XDB login on the Oracle instance used as the repository for Data Integrator it worked!  Up came a directory listing in my browser representing the XML structure residing in and served by the Oracle database serving as the Data Integrator repository.  In finalizing the solution I wanted to now switch from using the Oracle XDB feature in our DI repository to using XDB via the other Oracle database residing on the same box so as not "disturb" the DI repository database.

In the course of researching this situation, I found several options to accomplish this switch.  These included such tactics as disabling the Oracle XDB features completely or changing ports for listening.  What I ended up  doing was leaving everything associated with the Data Integrator repository database as is including XDB enabled on 8080.  I then reconfigured the other Oracle database to listen for the purposes of XDB on 8081 instead (I alse changed the default FTP port from 21 to 2111).   I found the details of these changes on the Internet will put up some links in a follow-up post.

The good news is all seems to be working under our control and now we can put Oracle XDB through a few paces without impacting the Data Integrator installation on the same box.  Mystery solved.  At least until the next one!

Last week I was working with Apache Tomcat off and on, building an understanding of the basics of its operation.  One of the platforms I work with regularly is BusinessObjects' Data Integrator and the version we are using runs on a HP-UX box.  The web administrator for Data Integrator uses Tomcat as its application framework (Oracle is the main Data Integrator repository).

I wanted to get a better understanding of how the web admin is enabled on the back-end and that led me to Tomcat.  As usual I began looking for and reading all the Tomcat related information I could find.  I was quickly scratching my head trying to understand all the pieces that made both Data Integrator and Tomcat do the work they together.  Honestly, I found the "documentation" on the Apache Software Foundation website less than beginner friendly.  A delivery from Amazon of the O'Reilly Tomcat book didn't help much at first.  Both the Apache website and the O'Reilly  book seemed like they will be more useful as my knowledge increases.

But I've stuck with it and have made some good gains in my understanding in the process.  I was able gather from various readings that there are both sample and administration applications as part of a standard Tomcat installation, if they haven't been removed.  Notice the qualifications of "standard" and "haven't been removed".  I wouldn't necessarily call the Data Integrator setup of Tomcat standard.  I don't think it's particularly unique but it's (DI) a commercial, production distribution and I shouldn't expect that everything would be as it described in available documentation on the Tomcat platform.

Fortunately I have worked with Java technology in the past and have a working knowledge of various implementations, including servlets and JSP.  But I couldn't find the Tomcat samples or Manager webapp on our installation.  With some persistence I was able to piece together the required elements and eventually get the Manager app working in our Data Integrator environment.  Let me say that there wasn't any problem with Data Integrator; I merely wanted a better understanding of how Tomcat was used with Integrator and what options we may have for integrating or enhancing other development within the existing DI/Tomcat infrastructure.  I figured implementing the standard Manager webapp, since it wasn't there and it's part of the Tomcat distro, would be a useful exercise.

This has proved to be the case and I now having the Manager webapp working within our Data Integrator installation and the DI portion has not been adversely affected.  I assumed (hoped) this would be the case, as from a "textbook" perspective there doesn't seem to be any reason why this wouldn't be possible.  Not only should it be possible, my understanding is that running this different components (Data Integrator and the Manager webapp) withing the Tomcat framework is exactly what Tomcat was designed to do.  I would guess that Tomcat experts would say, "Of course; what's your point?"  Well I've already mentioned that I wanted to see how this works and in my usual fashion that meant doing it, that's all.  In hindsight not particularly difficult, but then again if you haven't done it before it can be challenging as well.

After successfully converting dozens of tables in an Oracle data warehouse from uncompressed to compressed without so much as a single glitch, several problems cropped up last week.

We are also using BusinessObjects (now SAP) Data Integrator for ELT operations and have been for over 2 1/2 years also without much issue generally speaking.  The tables in question were just recently converted to compressed and then we started seeing a relatively small number of problems with our Data Integrator loads.

The problem presented itself when we truncated the compressed tables and then subsequently reloaded them.  We then seemed to be having problems with duplicated data.  No errors were given with truncating the tables in question.  It wouldn't seem possible that we'd get a problem with duplication after truncating a table, but nothing is taken for granted.  There was apparently no duplication in the source data used to popuate these tables.

When we switched the tables involved that we had changed back to the original attribute of uncompressed, the problem immediately went away.  And though we have converted dozens of tables to use compression as mentioned, all of which part of Data Integrator load jobs, we've only seen these issues involving a few converted tables.

We still have more investigation to do on this and we're not entirely certain it's related to the use of compression, but it sure seems more than just coincidence.  Also I don't know if I have the facts stated 100% correct but I think my details are pretty accurate nonetheless.  We'll see what we can determine but I'm still satisfied the use of compressed tables is reliable and worth the space we have reclaimed.