Wednesday, July 27, 2011

Debug mode for MOS

Had an SR in which I learned about a debug mode for FLASH MOS (tried it in HTML, no go *grin*). Hold down the Control key and click on the Oracle My Oracle Support Logo in the upper left-hand corner:


Here is a short video, using Oracle's recommendation of CamStudio:

Wednesday, July 20, 2011

MOS 5.3: Search and Knowledge

After my initial MOS 5.3 post, I received some most excellent feedback and an opportunity to speak with Jan Syssauw. He was exceedingly gracious and honored me with a 1-hour phone conference in which we covered many different aspects of Search. In response, I thought it would only be appropriate to follow up here.

I have a lot of respect for Jan (pronounced Yan). He is a very good listener and has a very professional demeanor, accepting my comments (and sometimes criticism) without belying how personally he cares for his team and his product. And yet he does - Jan very much wants Search and the KB to be an awesome experience, so I appreciate that he is reaching out to see what at least one user thinks.

In retrospect, I think I may have disproportionately emphasized certain features or areas by leveling the playing field (from my point of view) and perhaps giving the impression that I thought all areas were equal and required an equal degree of attention. Let me say that I wish to correct this. :) And I'll focus first on what I think the big issues are, and leave the small things for another time.

Search Suggestions

First and foremost, "Search Suggestions" are a huge improvement. I initially called these "type-ahead" or "autofill", but the proper term (so I have learned) is "search suggestions". I really like the direction they are going with this. We talked about some of the nuances; for example, the issue with versions I raised in my initial post (typing "Oracle server ent" gives me several suggestions, but none of them are 11g and only one is 10g). The reason Jan gave for this was that there are lots of suggestions that kind of fall off the end of the list - if I set my preferences to see only 5 "search suggestions", then even more of those suggestions would fall off the list. Jan hopes that the more popular and more relevant suggestions will start to percolate to the top of the suggestion list. I still find it odd that Oracle 7.3 would supercede 11g.... *grin*

Need for Speed

Ok, so FLASH is REALLY slow for me, and I REALLY do not like the way FLASH was implemented for MOS. However, setting FLASH aside and focusing on Search, I was struck again by a need for speed. I think these are both very related, but I am going to break it into two different paragraphs to help spell it out.

First, there are times when typing in a somewhat generic term (ie, "install"), I have to wait several seconds (I counted 9 seconds to retrieve 1.1 million records). What about first_rows_k?? *grin* Seriously, Oracle's flagship product is supposed to be the database, and they call themselves the "Information Company". While 9 seconds out of my life is peanuts, other companies (do I need to spell it out?) don't have you sitting around for a few seconds wondering how your fingernails got a little dirty. I have even filed a bug on a case where you can do a search on nothing and the backend server times out - as if I wanted to see every single document. *laugh*

Second, Jan mentioned a couple times when we were talking about setting limits for maximum number of Search Suggestions and Number of Records to Display that setting the limit higher would negatively impact performance. How much? If I want, let's get extreme, 100 search suggestions and 1000 results per page, will my search slow down to 10 seconds, or 10 minutes?

Search needs to be fast! Freakishly fast. They need Jimmy Johns to come and tell them about fast! *grin*


Preferences, options, filters....

This one is quite tricky, and I partially found the right words when I spoke to Jan and I hope I communicated it clear enough. I and other customers have asked for more options, more flexibility, more controls. And by golly, Oracle delivered! We know have Preferences, various ways to set filters via PowerView, Product, Task/Intent, etc. In my opinion, the way these options are presented to the user are a bit confusing. Take for example PowerViews. PowerViews came out a little while ago (MOS 5.1, 5.2?) and thus are a separate form of customization than, say, refining a search via the Knowledge Browser options (by Product, Task, etc). If I set a PowerView to view only Solaris platforms, that affects all my searches. So even though the PowerView summary is at the top of the page, I often forget I have a PowerView set and I rather puzzled when various search results are filtered out. Imagine trying to look for patches specific to Red Hat if you have a PowerView set for Solaris. :)

I do think it is awesome that Oracle is giving us, the customer, this much flexibility and control, and I applaud that. I only hope that we can work together to make the presentation of that power and flexibility more aesthetic and easier to grasp. In some ways, it is almost like going from a FPS game with simple controls to a Flight Simulator with tons of controls. I think this issue will be a matter of acclimation and user training for advanced users that want to take advantage of such knobs and levers; all other users wanting a quick, "just get me the results like Google" approach should not have to worry about these extra bells and whistles. Perhaps one possible way to approach this is to have an "Advanced Search" preference, and consolidate all such advanced features in one area. Maybe. At the very least, have the one preference to toggle the kitchen sink on and off. :)


And some lower-priority items

Some other low-hanging fruit we can take advantage of might be to reduce confusing by consolidating the use of "Task" and "Intent". My guess is that a bunch of folks got together and brainstormed on what to call this, but could not come to a final decision without offending others, so they compromised. :) So I will say, as one customer, just call them Tasks. Forget about "Intent". In my mind, "intent" sounds like you want to practice your ESP or something. But you knew that before I typed it, right....

Redo the way an article is displayed. For a while now, the articles slide in from right to left, squeezing the left-hand side. Either get rid of that effect, or make it happen in .4 seconds max.

I like the "intent" (tongue in cheek) behind the "auto-detection" features. When on the conference with Jan, he demonstrated what this auto-detection is currently doing; for example, if you type "install" in the search box, you wait a few seconds than get a screenful of suggested articles. At the top of the list is a box asking you to clarify your intent (hmm... there is that word again), which can be used to help filter results. A similar thing happens if you search for database (refine by product, for example). This is somewhat similar to what we developed at "The Bridge".

Thursday, July 14, 2011

MOS 5.3

re: Document 1326487.1.  My Oracle Support Release 5.3 Features

I knew that new features/enhancements were coming to MOS this past weekend, and I was looking forward to seeing what they were. I had heard that Search was improving. So when I read this Release Document, I was just a little underwhelmed at the level of detail. The "Release Highlights" all sound good, but I wanted meat. And then just today I realized that the details are actually in another document, My Oracle Support Release 5.3 Detailed Benefits Table (Doc ID 1329876.1)

More details in this document. But still not quite what I was hoping for. None-the-less, I decided to look into them.

Please note, I am a bit cynical in my comments. I realize a lot of hard work went into these enhancements, and there is a ton of "stuff" going on behind the scenes that I do not know about.


[EDIT: Egads, Blogger really mucked up my tables!!! Holy off-the-page, batman!]


Knowledge preferences
Customize your preference page
Knowledge preferences page on the Settings tab allows you to customize your knowledge settings

There are only 4 things you can customize, "Search Term Suggestion" (and associated number, max of 10), "Search Intent Clarification", "Search Result Set Length" (with a measly max of 25), and default "Searched Sources". That's it. I am disappointed. The "Searched Sources" only has two options, either the KB or ALL. Some preference. Better than nothing, I guess.



Support Identifier ManagementAbility to delete multiple Support Identifiers at onceNew window added for sizing control to change number of rows displayed for Support Identifier and managed users





meh. Could be helpful for organizations with really large numbers to manage. Glad we don't.




Search ImprovementsSearch Term Suggestion; search box has a type-ahead feature offering search suggestions using our knowledge base dictionaryFaster search results







Here we go, the fabled Search Improvements!
I have put this to the test already and it does help a bit. The only downside is that it is only available in FLASH (of course), since FLASH slows things down for me.




Message Center on the DashboardThis new feature consolidates all the pending user action for Automated Service Request, Configurations, and new user Support Identifier requests





I had to look carefully for this - it is really small, and in the upper right-hand corner, and often occluded by the annoying green slider message box that I was told was going away. Automated Service Requests (ASR) are the domain of Sun products, thus not relavant for me personally. I could care less if a "pending user request" popped up, since I get mailed about them anyway. I am actively looking for a way to make this "Message Center" go away. Not sure why this is under the "Search Improvements" section.



Auto-detection of product namesQuickly refine your search results for a product



This is still a bit buggy. When I type "Oracle server ent", I get 8 products, 6 of which are more than 6 years old, nothing for 11g or 9i, only one for 10g. "Quickly refine"?!? Or maybe I am simply using it incorrectly - not really obvious to me.


Auto-detection of common tasks and intentsQuickly refine your search results based on selected task 



Not exactly sure how this works. Obviously, not relevant for the search bar on every page, since there is no "task" selected. And it seems you can only select a "task" once a product is selected, no way to skip right to it. When I did look for a "task", the "Customer Recommended" document was #23 out of 25. I see that it is sorted by date, but not finding any way to sort by, say, reverse date, or even "Customer Recommended" for that matter. Or even by rating. The "Refine Search" bar on the left is really slow.



'Customer Recommended' indicator on search hit listBenefit from the feedback of other users who liked knowledge base documents




Mentioned above. It is not clear exactly how a document gets the fabled rating. Seems similar to the "like" button.



More comprehensive and intelligent search tipsAdditional guidance to get the most out of the search engine



Not much to say about this one - I'll have to take their word for it. The "type-ahead" auto-fill thing is probably the most visible aspect, and I like it so far.



More search sources in the form of additional product documentationExpands the knowledge base of possible solutions




So, now, I can search for JDEdwards and Sun documentation. Umm.. yippee?



Ability to set default search source to "All Sources" (in preferences)Expands the knowledge base of possible solutions




Mentioned above, but don't oversell it. A small step in the right direction. :)



Browse / Navigation ImprovementsRedesigned Knowledge Management tab with new "Browse Knowledge" regionEasier to find content that solves your problem





Not seeing how this works. Thus not easier at all. How do I do a search for a specific product and a specific intent with my keywords? Seems like when I enter keywords in the search bar, it completely ignores what I selected for product and intent. Where are the enhancements?



Direct navigation to "premium content" using task, product and release refinementEasier to find content that solves your problem




Easier, eh? Still can't find it. I must be really stupid. :)



Removed product browse tree (in favor of Knowledge Guidance)Easier to find content that solves your problem





Umm... they just replaced "tree" with "hierarchy". And moved it into a drop-down button. Oh wait, they had that before as well.




Health RecommendationsAbility to set lifecycle (Production, Stage, Test, Development) of multiple Systems at once Easier to maintain and manage Systems and Targets






Interesting. I guess that is a good thing, and it seems to work. Gotta love the little warning message (which should be removed, imo):
"The change will take approximately 2 seconds to complete. A message will inform you of the final result."


I am not going to comment on the other 5 sections of "Health Recommendations" because I do not use them. For folks that do, I can see how these might be good things, but given what I have see so far with other "enhancements", I have to question how much of an improvement this actually is.



Guided ResolutionTask-based Advisor helps to guide through a series of steps in a taskHelps you solve problems based on the process or workflow




I have no idea what this means. Where does one find this "Task-based Advisor"? Going to have to skip this whole section because I have no idea what they are talking about. It does recall to mind the old Libraries they used to keep. Again, sounds like a good idea, just not really obvious. Or maybe I really am dumb.



My Oracle Support CommunityNew user profile enhancementAllows user to enter biographical or specialized information and the ability to control emails sent from Oracle.






Not finding anything I want to fill out in this "profile enhancement". Provide my education?? Manage Certification logos? Gah! And the only knob for "controlling emails" is a radio button that says "receive emails based on your community subscriptions". Not what I was expecting at all.



Ability to upload documents, pictures, and videosAdd information in the format that communicates your question or situation clearly 




This sounds like a good thing - I did not realize you could not do that before. But I only have 8 Community points, so....



Allow users to view and manage the documents they subscribe toHelps to manage solutions so you can solve problems faster




This made me laugh.






General notes:

  • still see annoying green slider box in upper right.
  • when displaying a document, still see the annoying "slide left" effect that takes a couple seconds.
  • Good ideas and intention. Disappointed with the implementation.
  • still hate the lack of any decent "back" functionality (a FLASH thing). Clicking through breadcrumbs erases options I might have selected earlier. (Big problem with search!!)
  • new tab for "Advanced Customer Services". Really, an advertisement?!? How do I delete that?



To Do:

  • Learn more about Health Recommendations and Guided Resolutions. Dare I have high hopes?

Friday, June 17, 2011

Bug 11858963: optimization goes wrong with FIRST_ROWS_K (11g)?

At the beginning of March, I noticed some very odd things in a 10053 trace of a problem query I was working on. I also made some comments on Kerry Osborn's blog related to this matter. Oracle Support turned this into a new bug (11858963), unfortunately an aberration of Fix 4887636. I was told that this bug will not be fixed in 11gR1 (as 11.1.0.7 is the terminal release), but it will be included in future 11gR2 patches.

If you have access to SRs, you can follow the history in SR 3-314198695. For those that cannot, here is a short summary.

We had a query that suffered severe performance degradation after upgrading from 10.2.0.4 to 11.1.0.7. I attempted to use SQLT but initially run into problems with the different versions of SQLT, so I did the next best thing and looked at the 10053 traces directly. After a bit of digging, I noticed several cases where the estimated cardinality was completely off. For example:


First K Rows: non adjusted N = 1916086.00, sq fil. factor = 1.000000
First K Rows: K = 10.00, N = 1916086.00
First K Rows: old pf = 0.1443463, new pf = 0.0000052
Access path analysis for FRRGRNL
***************************************
SINGLE TABLE ACCESS PATH (First K Rows)
Single Table Cardinality Estimation for FRRGRNL[FRRGRNL] 
Table: FRRGRNL Alias: FRRGRNL
Card: Original: 10.000000 Rounded: 10 Computed: 10.00 Non Adjusted: 10.00



So, the idea behind FIRST_ROWS_K is that you want the entire query to be optimized (Jonathan Lewis would spell it with an "s") for the retrieval of the first K rows. Makes sense, sounds like a good idea. The problem I had with this initial finding is that every single rowsource was being reduced to having a cardinality of K. That is just wrong. Why is it wrong? Let's say you have a table with, um, 1916086 rows. Would you want the optimizer to pretend it has 10 rows and make it the driver of a Nested Loop? Not me. Or likewise, would you want the optimizer to think "Hey, look at that, 10 rows, I'll use an index lookup". Why would you want FIRST_ROWS_K to completely obliterate ALL your cardinalities?

I realize I am exposing some of my naivete above. Mauro, my Support Analyst corrected some of my false thinking with the following statement:

The tables are scaled under First K Rows during the different calculations (before the final join order is identified) but I cannot explain any further how / when / why.
Keep in mind that the CBO tweak -> cost -> decide (CBQT is an example)
Unfortunately we cannot discuss of the CBO algorithms / behaviora in more details, they are internal materials.
Regarding the plans yes, they are different, the "bad plan" is generated with FIRST_ROWS_10 in 11g

The "good" plan is generated in 10.2.0.4 (no matter which optimizer_mode you specify, FIRST_ROWS_10 is ignored because of the limitation) or in 11g when you disable 4887636 (that basically reverts the optimizer_mode to ALL_ROWS).
Basically the good plan has never been generated under FIRST_ROWS_10 since because of 4887636 FIRST_ROWS_10 has never been used before



I still need to wrap my head around "the limitation" in 10.2.0.4 and how we never used FIRST_ROWS_K for this particular query, but I believe that is exactly what Fix 4887636 was supposed to be addressing.

Here are some of the technical details from Bug 1185896:

]]potential performance degradation in fkr mode
]]with fix to bug4887636 enabled, if top query block
]]has single row aggregation
REDISCOVERY INFORMATION:
fkr mode, top query block contains blocking construct (i.e, single row aggregation). Plan improves with 4887636 turned off
WORKAROUND:
_fix_control='4887636:off'

I assume fkr mode is FIRST_ROWS_K, shortened to F(irst)KR(ows). The term "blocking construct" is most interesting - why would a single row aggregation be labeled as a "block construct"?

Also, this was my first introduction to turning a specific fix off. That in itself is kinda cool.

Monday, May 16, 2011

alert.log appears not be updated

After a few days of spinning my wheels and subjecting the poor recipients of oracle-l to multiple posts, I have identified an issue in Oracle code that I believe needs to be looked at.

First, some background.
We are running Oracle EE 11.1.0.7 on Solaris 10. We also have a job that occasionally bzips (compresses) the alert.log. The logic in the job is supposed to check if the file is actively being written to before zapping it, but by pure chance (so it would seem), in this particular case the alert.log was still open by the database when the file was scorched. This led to the appearance of the alert.log not receiving any more updates from the database. We attempted to bounce the database which had no discernible effect. I also changed the diagnostic_dest, which caused us to go from slightly strange to absolutely bizarre, and what opens the door for the rest of this post.


What I found
After changing diagnostic_dest several times, posting on oracle-l, the Oracle Community forums and playing tag with an Oracle Support Analyst, and doing lots of truss commands against sqlplus, I started to focus on this result from truss:
access("./alert.log", F_OK)              = 0

Now, you may notice that this "access" command is saying that the file in question ("./alert.log") is legit. This caused no small amount of head-scratching. I got the same results no matter which directory I ran the commands from. In my system, I only had two files with this name, one in $ORACLE_HOME/dbs and one in $DIAG/trace. Neither were actively updated by the database. It was not clear to me, at first, that Oracle was finding one of these log files. Especially since it never did anything with it. I searched file descriptors in /proc/*/fd and found nothing. I even grepped keywords from all text files looking for strings that should show up in this particular alert.log.

For the life of me, I could not figure out what directory ./alert.log was in. When I compared to other databases, this same access always returned Err#2 ENOENT. So I knew this must be key, but not sure exactly how. On a whim, I decided to delete the alert.log in $ORACLE_HOME/dbs. Lo and behold, the problem seemed to go away magically.

The BUG
So here is the root problem, in my opinion. The Oracle code line is looking for $ORACLE_HOME/dbs/alert.log, but completely fails to write to the file if it is found. Instead, the branch simply exits out. How is that helpful?

In retrospect....
I believe when I changed diagnostic_dest to a non-existing directory, Oracle automatically created alert.log in $ORACLE_HOME/dbs. I guess I learned a few things from this. :) Also, I learned a few tidbits along the way. One can use KSDWRT to write messages to the alert.log. Dan Morgan's library (still hosted by PSOUG) shows this. Also learned a little more about truss and dtrace as I was researching this issue.

Now the hard part; convincing Oracle that this is a problem and needs to be corrected.

Tuesday, May 10, 2011

"The Bridge": Day 3 (part 2)

I have received some pictures (not all, but most the important ones).

First, a recap of Day 2:
Our "realistic" picture evolved a little bit; Ahjay added some grouping tags ("WHERE", "WHAT") which we incorporated from there on out.


And here is what our OBJECT list finally looked like; complete with attributes and verbs:



Day 3
Hard at work.


After hashing things out in the morning, we finally had something akin to a prototype forming at our fingertips.

I really struggled with the overall complexity; I wanted simplicity. As a compromise, we worked very hard to make as much optional as possible, attempting to capitalize on pre-filled defaults and "quickfill" options, trying to use the technology and data that should already be available to reduce user interaction. For instance, if the user might be presented with the most recent Products at the top of one's list. Or setting your default QuickFill option (Previous SR, Profile or OCM) in your global Preferences. You will see, also, at the top left blue stickies for "Support Recommended" and "Product specific tips"; these are to be dynamically populated as you type and fill in information - the more information the user provides, the more relevant and specific the search becomes. I do not have any pictures, but on one of our white sheets we put in a meter as a gimmick to relate how more information upfront helps the user and the analyst focus on the problem (akin to the Password Strength Meter).

Near the end of the day, our final draft prototype was looking like this:


Again, you can see how "insta search" is being populated in the right-hand side, hopefully not too distracting, but also hopefully to be filled with information that would perhaps prevent an SR or guide a customer down the right path. Again, we are assuming huge improvements to Search. :)  This picture also demonstrates one possible "multi-screen" approach, trying to cram in as much as possible "above the fold". I argued for the "one-screen" approach, but compromised and suggested that a Preference be added to allow either one-page or multiple pages.

Another thing that might be slightly less obvious is that we are trying to keep the big picture in mind, or "tell a story" as Kelli put it. We are trying to describe a problem, which has a beginning (ie, the environment), a middle or body (the Description) and an ending (optional files, template questions, further elaboration, etc).

In the end, it still feels like way too much complexity to me. I noted earlier that I really want to talk to a human to route the issue (which obviates the whole "Category" mess). I do not mind filling in all the technical details, but what if you had a "Contact Analyst" button that, like Amazon and many other companies, auto-dialed you (the user) and attempted to get a IHUB person on the phone asap? Yes, I realize from Oracle's standpoint this is impractical. But does anyone else want that?

It will be interesting to see what comes out of this project. I think I am excited. The workshop itself was definitely very productive, eye-opening and an awesome experience that I am fully thankful to Oracle for.

Before we all parted ways, we did get a group photo. Say "Cheese!"

Thursday, May 05, 2011

"The Bridge": Day 3 (part 1)

Still no pictures yet, so this is Part 1 of Day 3.

Day 3 was crunch time; by 5:pm we were aiming to have a working prototype. Because we expanded our scope (rather significantly) and spent so much time on tangential (but very important and sometimes relevant) details, the idea of getting a working prototype seemed rather dubious. But I think we did it. To a degree.

Picking up where we left off, we started to tackle the actual UI design itself. We had already done a lot of work on Search, so we needed to focus on the SR part of it. I came in a little earlier and drew up my own mock ups - they are horribly cluttered, but I personally think they are kinda cool. :) Basically, my mockup capitalizes on the vast similarities between Search and Creating an SR; providing keywords (ie, title), a product (and version) and you can start going to town. Category is a bit tricky, and I will cover it a little more in the last paragraph, but if you can nail down Category you can potentially narrow down your Search (called "Task Intent") rather dramatically and better yet, you are primed to punch in and route an SR. So why not do both in parallel? Maybe even on the same screen. You start filling in information, and in one pane you start seeing search results aggregated by facets (like what Advanced Search does now, but much more dynamic and insta-search), while at the same time your "Create SR" button lights up. And maybe even a "Post to Forums" button. I briefly argued for this approach, and I readily admitted that the huge downside is that the screen gets very cluttered very fast. I think we adopted a hybrid (eg, compromise), where the "Related articles" shows up insta-matically in a somewhat unobtrusive region floating off to the side.

We did a couple of usability tests; frankly, I think we need specific "Usability Test" training to learn how to do these better. :) I was not entirely satisfied with the particular way we approached this topic. But the good news is that we discovered many holes in our current prototype. Late in the day, we voted and started to tackle some of the more critical (or easy-to-fix) issues. Near the top of that list was whether or not to display the entire SR Creation process as one page or multiple pages. Again, some were very concerned about cluttering the screen and wanted "screen-sized" sections. I want everything on one page. In the end I posited that the user should have a preference for how he/she wants to view this process. We will see what happens with that.

Actually, this topic consumed a bit of time. After we green-lighted the idea of multiple pages, we got to work going through several permutations of possible screen layouts. Again, I found it ironic that we kept coming back to a design that is very similar to what we have today in MOS. Granted, we are added a lot of behind-the-scenes features that auto-fills (and insta-searches) as much as possible - that is not to be overlooked. But our final "look and feel" does not diverge much from the current design, in my opinion. In fact, if I count correctly, our final design may actually look more complicated. It is hard to say without having a real GUI to step through. Even though it looks more complicated, we are actively working to allow the user to input as little as possible to get the SR filed.

I have mentioned this previously, but it bears repeating. We were very much biased by the current implementation. In some ways, we spent a huge chunk of time trying to "fix" and patch current brokeness, instead of redesigning from the ground up. This is not to say we did not think out of the box (or at least try to).  And right now as I type this, I cannot think of one single "out of the box" new thing we pushed. Maybe I am simply tired and not remembering well.

Another point of discussion that came up, and in retrospect I wished we spent more time on, is the current super-criticality of "categories". Currently, SRs are routed based on the sub-category (or category if no sub exists). These are currently filtered by which product one chooses. In our experience, choosing the most appropriate sub/category is often tedious and seems like a relatively useless step from the users point of view. We briefly talked about driving the sub/category off keywords in the Description field, and to be done in the "insta-search" way (you start typing, and the list of possible sub/categories to choose from grows smaller). But the bigger issue, in my opinion, is all about the routing in the first place. Oracle has placed a lot of emphasis on building automated logic to get the SR to a specialist team. I have a problem with that, at least how it is done currently. In my personal "Bleu Sky" vision (Day 1), I created a big easy "Create SR" button, with no requirements whatsoever. How the heck is that any good? Well, think about it, what happens? Rather, what if you changed the button to say "Chat with a human being"? By the end, we made comparisons to various other companies (ie, Amazon) that allows you to fill in call-back information, a computer actually calls you 1 second later, and then attempts to connect you to a live person. I love that concept!! As you can imagine, the managers and directors and support representatives at the meeting hated that idea. :) Yes, currently, it is hugely impractical - the IHub would be drowned to oblivion. Currently. But if we are thinking Utopian thoughts.... There are other ideas to simply routing. For instance, drastically reduce the number of routes. How? Well.... we didn't talk about that, yet. :)