Monday, May 16, 2011

alert.log appears not be updated

After a few days of spinning my wheels and subjecting the poor recipients of oracle-l to multiple posts, I have identified an issue in Oracle code that I believe needs to be looked at.

First, some background.
We are running Oracle EE 11.1.0.7 on Solaris 10. We also have a job that occasionally bzips (compresses) the alert.log. The logic in the job is supposed to check if the file is actively being written to before zapping it, but by pure chance (so it would seem), in this particular case the alert.log was still open by the database when the file was scorched. This led to the appearance of the alert.log not receiving any more updates from the database. We attempted to bounce the database which had no discernible effect. I also changed the diagnostic_dest, which caused us to go from slightly strange to absolutely bizarre, and what opens the door for the rest of this post.


What I found
After changing diagnostic_dest several times, posting on oracle-l, the Oracle Community forums and playing tag with an Oracle Support Analyst, and doing lots of truss commands against sqlplus, I started to focus on this result from truss:
access("./alert.log", F_OK)              = 0

Now, you may notice that this "access" command is saying that the file in question ("./alert.log") is legit. This caused no small amount of head-scratching. I got the same results no matter which directory I ran the commands from. In my system, I only had two files with this name, one in $ORACLE_HOME/dbs and one in $DIAG/trace. Neither were actively updated by the database. It was not clear to me, at first, that Oracle was finding one of these log files. Especially since it never did anything with it. I searched file descriptors in /proc/*/fd and found nothing. I even grepped keywords from all text files looking for strings that should show up in this particular alert.log.

For the life of me, I could not figure out what directory ./alert.log was in. When I compared to other databases, this same access always returned Err#2 ENOENT. So I knew this must be key, but not sure exactly how. On a whim, I decided to delete the alert.log in $ORACLE_HOME/dbs. Lo and behold, the problem seemed to go away magically.

The BUG
So here is the root problem, in my opinion. The Oracle code line is looking for $ORACLE_HOME/dbs/alert.log, but completely fails to write to the file if it is found. Instead, the branch simply exits out. How is that helpful?

In retrospect....
I believe when I changed diagnostic_dest to a non-existing directory, Oracle automatically created alert.log in $ORACLE_HOME/dbs. I guess I learned a few things from this. :) Also, I learned a few tidbits along the way. One can use KSDWRT to write messages to the alert.log. Dan Morgan's library (still hosted by PSOUG) shows this. Also learned a little more about truss and dtrace as I was researching this issue.

Now the hard part; convincing Oracle that this is a problem and needs to be corrected.

Tuesday, May 10, 2011

"The Bridge": Day 3 (part 2)

I have received some pictures (not all, but most the important ones).

First, a recap of Day 2:
Our "realistic" picture evolved a little bit; Ahjay added some grouping tags ("WHERE", "WHAT") which we incorporated from there on out.


And here is what our OBJECT list finally looked like; complete with attributes and verbs:



Day 3
Hard at work.


After hashing things out in the morning, we finally had something akin to a prototype forming at our fingertips.

I really struggled with the overall complexity; I wanted simplicity. As a compromise, we worked very hard to make as much optional as possible, attempting to capitalize on pre-filled defaults and "quickfill" options, trying to use the technology and data that should already be available to reduce user interaction. For instance, if the user might be presented with the most recent Products at the top of one's list. Or setting your default QuickFill option (Previous SR, Profile or OCM) in your global Preferences. You will see, also, at the top left blue stickies for "Support Recommended" and "Product specific tips"; these are to be dynamically populated as you type and fill in information - the more information the user provides, the more relevant and specific the search becomes. I do not have any pictures, but on one of our white sheets we put in a meter as a gimmick to relate how more information upfront helps the user and the analyst focus on the problem (akin to the Password Strength Meter).

Near the end of the day, our final draft prototype was looking like this:


Again, you can see how "insta search" is being populated in the right-hand side, hopefully not too distracting, but also hopefully to be filled with information that would perhaps prevent an SR or guide a customer down the right path. Again, we are assuming huge improvements to Search. :)  This picture also demonstrates one possible "multi-screen" approach, trying to cram in as much as possible "above the fold". I argued for the "one-screen" approach, but compromised and suggested that a Preference be added to allow either one-page or multiple pages.

Another thing that might be slightly less obvious is that we are trying to keep the big picture in mind, or "tell a story" as Kelli put it. We are trying to describe a problem, which has a beginning (ie, the environment), a middle or body (the Description) and an ending (optional files, template questions, further elaboration, etc).

In the end, it still feels like way too much complexity to me. I noted earlier that I really want to talk to a human to route the issue (which obviates the whole "Category" mess). I do not mind filling in all the technical details, but what if you had a "Contact Analyst" button that, like Amazon and many other companies, auto-dialed you (the user) and attempted to get a IHUB person on the phone asap? Yes, I realize from Oracle's standpoint this is impractical. But does anyone else want that?

It will be interesting to see what comes out of this project. I think I am excited. The workshop itself was definitely very productive, eye-opening and an awesome experience that I am fully thankful to Oracle for.

Before we all parted ways, we did get a group photo. Say "Cheese!"

Thursday, May 05, 2011

"The Bridge": Day 3 (part 1)

Still no pictures yet, so this is Part 1 of Day 3.

Day 3 was crunch time; by 5:pm we were aiming to have a working prototype. Because we expanded our scope (rather significantly) and spent so much time on tangential (but very important and sometimes relevant) details, the idea of getting a working prototype seemed rather dubious. But I think we did it. To a degree.

Picking up where we left off, we started to tackle the actual UI design itself. We had already done a lot of work on Search, so we needed to focus on the SR part of it. I came in a little earlier and drew up my own mock ups - they are horribly cluttered, but I personally think they are kinda cool. :) Basically, my mockup capitalizes on the vast similarities between Search and Creating an SR; providing keywords (ie, title), a product (and version) and you can start going to town. Category is a bit tricky, and I will cover it a little more in the last paragraph, but if you can nail down Category you can potentially narrow down your Search (called "Task Intent") rather dramatically and better yet, you are primed to punch in and route an SR. So why not do both in parallel? Maybe even on the same screen. You start filling in information, and in one pane you start seeing search results aggregated by facets (like what Advanced Search does now, but much more dynamic and insta-search), while at the same time your "Create SR" button lights up. And maybe even a "Post to Forums" button. I briefly argued for this approach, and I readily admitted that the huge downside is that the screen gets very cluttered very fast. I think we adopted a hybrid (eg, compromise), where the "Related articles" shows up insta-matically in a somewhat unobtrusive region floating off to the side.

We did a couple of usability tests; frankly, I think we need specific "Usability Test" training to learn how to do these better. :) I was not entirely satisfied with the particular way we approached this topic. But the good news is that we discovered many holes in our current prototype. Late in the day, we voted and started to tackle some of the more critical (or easy-to-fix) issues. Near the top of that list was whether or not to display the entire SR Creation process as one page or multiple pages. Again, some were very concerned about cluttering the screen and wanted "screen-sized" sections. I want everything on one page. In the end I posited that the user should have a preference for how he/she wants to view this process. We will see what happens with that.

Actually, this topic consumed a bit of time. After we green-lighted the idea of multiple pages, we got to work going through several permutations of possible screen layouts. Again, I found it ironic that we kept coming back to a design that is very similar to what we have today in MOS. Granted, we are added a lot of behind-the-scenes features that auto-fills (and insta-searches) as much as possible - that is not to be overlooked. But our final "look and feel" does not diverge much from the current design, in my opinion. In fact, if I count correctly, our final design may actually look more complicated. It is hard to say without having a real GUI to step through. Even though it looks more complicated, we are actively working to allow the user to input as little as possible to get the SR filed.

I have mentioned this previously, but it bears repeating. We were very much biased by the current implementation. In some ways, we spent a huge chunk of time trying to "fix" and patch current brokeness, instead of redesigning from the ground up. This is not to say we did not think out of the box (or at least try to).  And right now as I type this, I cannot think of one single "out of the box" new thing we pushed. Maybe I am simply tired and not remembering well.

Another point of discussion that came up, and in retrospect I wished we spent more time on, is the current super-criticality of "categories". Currently, SRs are routed based on the sub-category (or category if no sub exists). These are currently filtered by which product one chooses. In our experience, choosing the most appropriate sub/category is often tedious and seems like a relatively useless step from the users point of view. We briefly talked about driving the sub/category off keywords in the Description field, and to be done in the "insta-search" way (you start typing, and the list of possible sub/categories to choose from grows smaller). But the bigger issue, in my opinion, is all about the routing in the first place. Oracle has placed a lot of emphasis on building automated logic to get the SR to a specialist team. I have a problem with that, at least how it is done currently. In my personal "Bleu Sky" vision (Day 1), I created a big easy "Create SR" button, with no requirements whatsoever. How the heck is that any good? Well, think about it, what happens? Rather, what if you changed the button to say "Chat with a human being"? By the end, we made comparisons to various other companies (ie, Amazon) that allows you to fill in call-back information, a computer actually calls you 1 second later, and then attempts to connect you to a live person. I love that concept!! As you can imagine, the managers and directors and support representatives at the meeting hated that idea. :) Yes, currently, it is hugely impractical - the IHub would be drowned to oblivion. Currently. But if we are thinking Utopian thoughts.... There are other ideas to simply routing. For instance, drastically reduce the number of routes. How? Well.... we didn't talk about that, yet. :)

Wednesday, May 04, 2011

"The Bridge": Day 2

It is late and I am exhausted. And I have no pictures from today (/me looks at Richard Miller).


We started off on a good foot, having already taken a good stab at objectifying the tasks. We further hashed out all more objects, added more attributes, added verbs and relationships. Some objects were much easier than others; for instance, Product really only consists of a product name and version number in the scope of an SR. Yes, it is a "child" member of other objects.

In retrospect, we got bogged down in many areas, and sometimes it seemed like those areas were really minor and we were spinning our wheels. But the "spinning wheel" did demonstrate that even in our small group there is ambiguity and misunderstanding of core elements. For example, is a Primary Contact a subset of a "Contact Type"? Or is it different enough from other contacts (ie, Secondary, Manager) that it deserves it own types. One member argued quite vociferously that it is its own object because it is hanlded differently, like populated from a User Profile, while the others are not. After ripping up and redoing Contacts in various permutations, we finally decided on a single Contact object with various conditional properties and verbs.

There were other examples of the same thing, I just do not remember them off the top of my head. These little excursions took up a bit of time. On top of that, we also delved heavily into the Knowledge Base and Search, since we had decided to expand our scope the previous day. While much of our journey through this topic is quite useful in the context of filing and resolving an SR, it consumed time as well. So even though we had covered a bit of ground, Richard Miller declared that we were several hours behind in the late afternoon. :) I am not sure what that means for tomorrow.

Some very newsworthy things that came out of our session. I have not signed any non-disclosure agreements, but I do think the Managers want to keep a lot of new developments under wraps. So I'll go about it indirectly. We chatted up some more "Blue Sky" features as we discussed things we did not like about the current implementation. One of the key features to our new approach is using Search heavily at the outset of a possible SR creation process. I know, you are thinking this is a HORRID idea. But if Search were actually much improved (in terms of performance and relevance), we see this as being a huge boon. We could be wrong, time will tell. Basically, you have a form that provides an opportunity for the user to provide a ton of information. Much of the form is optional, but the idea is that the more you provide, the better the search results. Using ideas like Google's word-completion and instant results, and eBay's and Amazon's left-hand pane of refining and drilling-down, we explain how these kinds of features would significantly enhance the user's perception of Search by providing fast, dynamic feedback on the criteria entered. On top of that, the user may have a chance to save the search filters/results and shove all the pertinent information entered into an SR, or maybe even a Community forum post. Some of the above ideas have already been developed and we saw some simple demos. Like using quickfill and/or word completion in various areas. Very nice to see that they are already make in-roads in that direction.


I am particularly torn about the latest prototype GUI mock-up that our group has achieve so far. I claim my role, so I am not blaming everyone else. I say I am torn because the pages/screens that we "developed" today still look very busy and crammed full of things to fill out. It almost looks like we have merely re-arranged the existing SR fields that one normally fills out. I think the key importance in our approach today is that we are aiming for two things:
 - allow as many optional fields as possible
 - thus giving the user a choice between providing less detail and possibly a more vague search, or more detail and possibly a more accurate search

We are both assuming that search will be improved significantly, and providing ideas on how exactly to do that. The dynamic feedback mechanism is crucial I think, since it gives the user a good idea as to how many docusments are returned and how to refine it. I think. It looks good on paper right now. :)

Ok, that's it for me. I hope to procure more pictures tomorrow.

Tuesday, May 03, 2011

Do people really do this in real life?

"My name is Newton Sequeira and I am an Author Relationship Executive at Packt Publishing. Packt recently green lit a book on Oracle 11g R2 RAC Administration Cookbook and we are now searching for an author to develop the book.
I was reading through your blog and wondered whether you might be interested in this project?
Thanks for considering this proposal. I would appreciate if you could please let me know your views."



This simply scares me. Do publishers really approach potential authors this way? Believe you me, I am the very last person any publisher would want to be writing this particular book. Even if I were to complete such an ambitious project, I would be the laughing stock of the Oracle community. For at least a year. And based on what little I know of author-publisher relationships, the author endures crushing timelines and relentless editors and enjoys a very small fraction of the royalties. Yeah, awesome incentive there.


If you see my name on this book, don't buy it. Please.

"The Bridge": Day 1

Today we had a great session. Obvious introductions were first; another functional user who files a lot of SRs for Finance and HR modules, a couple of upper-level managers, a IHUB engineer and a front-end "user experience" ADF developer (was that too redundant?).

Honestly, I was a bit overwhelmed at first, having never done task flows in a group like this before. But I liked the concept. Our moderator/taskmaster Mitch is a good guy, and at times we tried his patience. :) We started off defining what we thought the "Big Picture" is - note how we labeled it "Create SR". At one point, one of the managers said, half-jokingly, "My god, what have we created?!?":


As you can see, we have lots of stickies. Mitch loves sticky notes (aka, post-it notes). We identified key processes and showed how they related to each other, and finally we marked it up with pink stickies for problematic areas. We also identified some "out of scope" topics. This first go was a really rough draft but provided a framework from which to build.

After we had the current picture in mind, Mitch asked us to dream about what we wanted it to look like. To dream a little. To think of a Blue Sky. Spelling errors (wrong to call them typos when you write them out? *grin*) were the trademark of the evening, but we pushed forward. The following picture is what the other user and I came up with:


I did the one on the left. I kept it really basic, because that is what Mitch indicated. And I wanted to emphasize how we need to keep the process simple and as fast as possible. The other user representative has a lot of experience filing SRs so has essentially figure out how to "game" the system to make it work fast. Included on the other side are a number of additional (and some optional) items, some of which overlap mine. I do not have a picture of the diagram the managers/developers came up with, but it wanted more forms and more questions answered. :) After getting it all down, we users then marked each step with how desirable it was (H = High, M = Medium, L = Low), and the developer group marked how feasible it was (H = Hard, E = Easy). The goal was to find as many highly desirable and easily feasible points as possible. I kinda think we did not pay too much attention to that. OH well. The next phase was putting these two (Realistic + Desirable) together. In the middle, we experience a Scope Changed because we started to see how important "Search" is to this process, and how we users much rather find existing information that solves our problem then filing an SR in the first place. Thus our Scope evolved into "Solving Problems" and this "Step 1" reflects how many of the things that could be used to initialize an SR could actually be pointed at the Knowledge Base. Like so:



Lest you despair (some consider KB to be a four-letter word), we had lots of talks about improving the KB search functions, and especially focusing on using the Advanced Search capabilities. This reflects the combination of our "Blue Sky" ideas - of simplifying the existing framework and trying to think of what is the bare amount needed to go search for information, while still providing plenty of robust functionality for power users who want to provide a ton of extra detail.

Here is a shot of us "in action" - you can briefly glimpse the chaos:



In the end, we also worked on "Step 2", which was the SR Creation portion of it, and discussed at length how these two steps play together, and how Sev 1 changes the ballgame a little (more often than not, if you file a Sev 1, you are not going to take the time to Search). With 30 minutes left of the night, we dove into objectifying the tasks. We merely scratched the surface, but I think we all felt it was significant progress.

I am certainly very impressed by this process. There are some obvious inter-group challenges when certain folks dominate the discussion, but overall we are making wonderful progress and I am very happy we are having these discussions. I only hope we are drastically pushing the managers and developers in a direction we will later regret. :) I am also struck by the complexity and the number of pieces involved. We very briefly touched on how OCM/EM play a role in providing data to the SR creation process, and we obviously tacked the bigger "purple elephant" of Search and the Knowledge Base (how many people are using it and finding what they need?).

And now I am mentally exhausted. And need to grab some sleep.