Confluence Error - Non Clustered Confluence: Database is being updated by another Confluence instance

So we had an interesting one earlier today - luckily it didn't take too long to troubleshoot, but it wasn't included in any of the Confluence support resource, so thought I'd share it here.

We use a single instance of Atlassian Confluence to host our internal IT Wiki - not critical but very important for the team as all of our process and procedures are stored here, along with all of our detailed infrastructure information.

The following message appeared when we tried to open the Wiki page:


-------------------------------------------------------------------

Confluence

You cannot access Confluence at present. Look at the table below to identify the reasons.

Type
cluster

Description Non Clustered Confluence: Database is being updated by another Confluence instance. Please see http://confluence.atlassian.com/x/mwiyCg for more details.Your server id is: XXXX-XXXX-XXXX-XXXX


Exception


Level
fatal

Time
local time

This page will automatically update every 60 seconds.

-------------------------------------------------------------------

-------------------------------------------------------------------

The online support suggests all sorts of troubleshooting but none of this was effective. Turns out, the transactions logs had grown too large and needed clearing down. Simples! 

I hope this helps reduce the troubleshooting time you spend on a similar issue! 

CISSP (part 3/3)

Passed!

I took the exam in London as planned, very nerve-wracking and took about 3.5 hours. Of the 250, I was sure that I'd answered about 40% correctly, another 40% I though should be OK and the last 20% I didn't want to ever see again! I revisited about 70 questions, second guessed myself a few times and then un-guessed a few of those. By the end of the exam, I wasn't confident of passing or failing - my eyes were starting to un-focus and my brain was fried...

You don't find out until you leave the examination hall if you passed or failed, and if you passed you only get a 'pass' - no score to say how well! If you fail, they'll let you know how you did on each CBK domain, allowing you to focus your future study.

The feeling of elation when they tell you that you've passed is great - I certainly didn't want to have to retake the exam. The next step is to get endorsed and I was able to get this done by a CISSP that I've worked with before who stands in good stead and was impressed with my work and knowledge when we worked together. 5 weeks wait before the confirmation email arrived and another 7 weeks before the certificate arrived in the post.

For those of you about to start studying, or currently preparing for the exam, here is a quick description of my study process:

• I used the CISSP exam study books for reference only - it's not written in an easy-to-read way and by the time you'd finish reading the entire book you'd have forgotten all of the content from the beginning anyway. Find a weak area and use these books to flesh out your knowledge. 
• The 11th Hour Eric Conrad book (know this inside out) - the content in this book came up more often than content from the official study book - but you'll need to know everything that's in there.  
• Skillset & CCCure for very simple and basic exam questions - they're not really anything like the real exam questions but tests your basic knowledge of all the domains. If you can use these to focus your study on your weak areas - I made sure I was hitting 80%+ for each exam I was doing before I booked the exam. 

The exam questions are not the simplest to decipher - make sure you take the time to read each question twice and then make sure that answer is the 'best fit' for the question as there may be more than one correct answer (one will be a better fit than the other though!). Examples here may include questions about encryption or a transmission protocol - there are more than one of each but one will be better suited as a solution to that specific question.

So take your time, know your stuff, be confident and I wish you the best of luck! 

Time to focus on my CPEs now - 40 per year to keep the qualification!

CISSP (part 2/3)

I recently wrote a short blog on my plans to attain my CISSP certification. I was planning to attend a week long revision course (remotely) and then plan my final few weeks of revision before taking the exam.

I've taken the week long course and feel much more confident than I did before hand on the whole CISSP CBKs. I've been working as an IS Admin / Sys Admin / Infrastructure manager / IS dogs-body for upwards of 10 years since I left uni and this experience is definitely given me a broad knowledge of IS security. My recent experience in setting up a Business Continuity Suite and the organisations Disaster Recovery plans was, I thought, going to ensure I knew most of the Risk Management section - but I was very wrong! The CISSP is all about what you have to do to identify, quantify and address risks and includes a raft of equations (not difficult) and possible models for getting this done.

Having spent a week looking at the CISSP exam, I'm getting the feeling that it is a lot of 'read-this-remember-that'. I wasn't under any illusions, I knew it wasn't very hands-on but thought it would be a little more in-depth than it is.

In a nutshell - there's a heck of a lot to learn (mile wide, inch deep) but my general experience in IS over the last 10 years, including 6 of those with a focus on BC, risk management and ISMS will definitely stand me in good stead. I'm not reading the huge exam books from cover to cover, but rather going through the smaller 11th hour and exam cram books - following that up with a focus on the things I know I'm not hugely knowledgeable about yet (e.g. Encryption). I'm slowly getting all of my practice exam scores up into the 80%+ range which is where I want them all to be before taking the exam. Which is booked for October, so watch this space!

VPN Error 691

Recently I've been having some issues with our VPN - every time I try to connect from home, I get and error 691 - 'Access denied because username and/or password is invalid on the domain'.

When I check the VPN logs, this error pops up as the connection is refused:
...vpn 0x01E=691 R=1...

We use a mobile One Time Password (mOTP) app to provide two-factor authentication, so I checked the secret, the pin and the time all match up on the VPN device and my laptop / phone - all looked good.

After much messing around I've managed to work out that the mobile phone is running about 40 seconds ahead of the VPN device - even though they're both pointing to Internet NTP servers... 40 seconds shouldn't make a difference, should it? Well it does. If I wait until then last 10 seconds of the current valid mOTP password and then connect, no problem. If I use a newly generated mOTP password, then it's a no-go.

So... 40 seconds makes a big difference in the world of VPNs!

CISSP

I've spent the last 10 years of my IT career working in and around Infrastructure, and everything else IT that doesn't get bundled with app development and testing. One area I've been slowly been gravitating towards has been Information Security and luckily I've been able to make sure I get plenty of experience in this area over the last 6 years or so.

We recently underwent a Data Safeguard Audit and I was able to use this an an excuse to kick of a body of work to review the entire security for our organisation. In the end, we did pass the audit but not after some seriously hard graft by implementing an ISMS according to the Gospel of ISO27001. 

On the back of this I've convinced the powers that be to fund my (remote) attendance at a CISSP review and exam prep course (of course no funds available for transport or accommodation). The course is less than £1,000 which is either great value or I'll get what we've paid for (I do hope it's the former). 

SO... I thought I'd share my experience of attempting to gain the CISSP qualification. I've been told it's not easy and requires some serious commitment - along with the stringent background checks to make sure you have your 5 years experience in IT security. 

I plan to read the Shon Harris CISSP Exam Guide (4th Ed.) and possibly the Official Guide to CISSP CBK(4th Ed.) over the coming weeks; (remotely) attend the CISSP Certification Exam prep course; create flash cards and post-its for all facts I'm not 100% on; download some study MP3s for the car and mobile; videos for my lunch breaks, evenings and weekends; and attend a CISSP study group once a week for 2 hours. All this without sacrificing too much of my social life?! I guess that will have to wait to be seen...

I'll keep the updates coming over the next few months - I am aiming to complete the exam within 3 months from now, so beginning of October if all goes to plan. I'll keep it to my personal experience - what works for me, what doesn't work, good resources, bad resources, etc., and feel free to let me know of any tips, tricks or useful links in the comments!

IS Policies - ISO27001

Data is becoming the biggest and most important asset that any company has, and it's important to ensure that your company data is properly managed. From an IT perspective, this can be managed through an Information Security Management System, according to ISO:27001. This International Standard is a great reference when putting controls in place, or mapping what controls you already have to industry standards.

We started this process by performing a GAP analysis - going through the ISO standard and mapping what controls, polices, procedures and processes we already had in place. We found that most of what we had didn't quite cover everything we had expected it to do. There was also a lot of controls in the Standard that we hadn't addressed.

Our next step was to update all of the policies we had to make sure that they did address all of the controls that they were designed for. We were able to amalgamate a lot of them and expand the ones we had left so that by the end we had fewer, but more detailed policies. We collated them all in a single handbook and then created an index spreadsheet that listed all of the policies, the owners, dates for review, changes etc. so that we had a quick reference document.

Once this handbook had been compiled, we went back to the ISO Standard and looked at the areas we had not yet addressed. Some more policies were created and added to the handbook. We also looked at all our our procedures and processes - standardised them all and if applicable we merged them into policies, or added them as appendices in a standardised format.

Once this had been all been done, we split the book into different sections and published them to the different areas of the company. One for the General Staff, consisting of all policies that applied to everyone; one to the HR department that covered... you guessed it, HR Policies; one to our internal IT staff that was pretty comprehensive and covered all of our controls, polices, procedures and processes; and the final section was published to suppliers and 3rd parties that we work with, consisting of policies in relation to SLAs, contracts and other external procedures that we had in place.

We review this ISMS every year at a minimum to make sure that the policies and controls still apply to us and our current use of technology. It has given the staff, management and IT team a lot more confidence in our security and controls and we follow these religiously.

The above description is a very, very brief outline of the work we carried out, which took several months to complete. It is a huge commitment and takes a very large amount of resourcing, but once in place and part of the culture, helps to increase your security posture and gives your company and others confidence in how you process and store your data.

Server 2008 R2 - server starts up but can't log in or use resources

Monday morning after a week off and surprise surprise, a call at 6am to say the system is down!

All servers are working - apart from one. The one that stores the user profiles (we discovered that our fail-over wasn't working, that blog will come later!). At the console we're met with the Ctrl+Alt+Del screen and are able to enter our credentials. We tried both domain and local users, but couldn't get further than either Please wait for the User Profile Service with the domain account or Loading Windows Personalisation with the local account. Luckily, with the local account, we could hit Ctrl+Alt+Del and bring up the Task manager.

With the Task Manager open you can select Services and see which services had started correctly and which ones were still in the Starting phase. For us, the Citrix MFCOM and IMA services were still Starting and gave us some clues as to what was happening. Cue a Safe Mode with networking restart and we were able to log in with the local account.

When we opened the Services.msc console we could see that since the last restart, the two services for Citrix mentioned above were now trying to log in with our Backup Exec account. A quick check on the other Citrix servers and we could see that it should be using the Network account. using the log on tab in the services properties, we were able to switch back to using the Network account.
*Quick Note* To change back to the Network account, open the service's Properties, choose the log on tab, click Browse and type in Network, then hit Enter - this will automatically change the account to Network. Remove the password in the boxes below this and Apply and close the properties box.

Complete this for both services and Restart the server. Hey Presto! Server and resources now work. All that's left is to investigate what (or WHO!) switched the log on account from Network to Backup Exec!

\\domain.com\namespace: The Namespace cannot be queried. Element not found.

After we managed to save our SAN disk and have it represented to the server after it had booted (I don't recommend it! Boot the server with the drives presented EVERY time), we had some unusual DFS issues. When we opened DFS Management to check one of the namespaces, it was showing:

\\domain.com\namespace: The Namespace cannot be queried. Element not found.

We could successfully open the folder on the server, but couldn't when trying to access as a network resource. We tried to remove the share on the folder using folder properties but it said we had to remove it from the DFS management console first.

After much playing around, we decided it had to be deleted. So... ADSIEdit > open the domain > CN=system > dfs-configuration > find the namespace you wish to delete and delete it. Open DFS management and remove the problematic share from view. Next - make sure you restart the server! We spent the next half an hour troubleshooting it - all it needed was a restart. This kicks DFS into play again and it catches up with itself.

At this point, open the folder properties, remove the option to share from advanced sharing and then set it all up again using the DFS Management console.

In all, quite a simple fix - but as the saying goes, it's easy when you know how!

EVA 4400 Cache Battery problems - EventID 1511 and 1521

It all started so well. At 7am on my day off (!) the phone went and I was told that no-one could log in successfully. Several people were managing to get in with a temp profile but couldn't access any local apps. Luckily most of our apps are run via Citrix, which was still available via Receiver and the Start menu.

Upon inspection, lots of EventID 1511 and 1521's appearing. The advice online - rebuild the corrupt profile. Well, this was happening for the entire office of over 100 people so that was an immediate no-go.

(***In a nutshell - if you get EVENTID 1511 and 1521, it usually means a corrupted profile. In our case, the SAN drive that contained the profile folders was not presented successfully to the server, so the profile folders were not available to the users as they tried to log in.***)

We're in the process of migrating from our old EVA 4400 to the latest and greatest HP has to offer. However, being 'in the process of' means we're running both systems in parallel. The disk from the new EVA was working fine. The disk from the old EVA just shows up for 5-10 minutes after the server boots up and then disappears from view. Nothing in Disk Management either. We removed the automatic updates that installed the night before, nothing. Swapped the Fibre Card over, nothing. Checked Command View on the management system and the EVA reported full health - apart from a Cache battery on Controller 1 having died. 

The Cache battery had died on us several months before, but had no side effects on the system. This time, for some strange reason, because the cache battery on Controller 1 had died, the MPIO routing from Windows died with it. It steadfastly refused to re-route the traffic to Controller 2 and as such MPIO on the server decided that it couldn't see the drive at all. Even using CV we couldn't get the system to fail over to the working controller 2. Very frustrating. Luckily we had a spare cache battery laying around.

After swapping the cache battery, bingo - the drive reappeared! Controller 1 came back up and the system went back to normal - almost. 

However... Read the next blog for the DFS fall-out issues! 

The requested operation cannot be completed because the terminal connection is currently busy processing a connect, disconnect, reset or delete operation.

I get this one regularly at work now when one of servers crash. It seems that their logon session remains connected and won't reset. There are plenty of solutions out there, but none worked for me until I found this one - so simple!
Run command prompt on the relevant server the user is having an issue with, and use the 'query process winlogon.exe'
This will show you all the winlogon processes on the server.
Find the one process without a SESSIONNAME (i.e. the only blank entry).
Make a note of the PID associated with that process.
Run Task Manager and click View menu > Select Columns... and choose PID (at the very top).
Now you can sort the list by PID number and find the winlogon process with the PID you are looking for.
Kill the winlogon process and Hey Presto, the user can now log in!

Hope this helps, took me a while to find!

Citrix... Again!

Halfway through Monday morning I get a note to say one of our users cannot access any Citrix published applications on one of our Citrix Servers (XenApp on Windows 2008). All other applications are fine, but anything on our second application server just gives the generic 'loading' message. As the day progresses, anyone else that tries to log on or create new sessions (key words: New Sessions) gets the 'loading' message. All users with open sessions are fine, programs keep working and they can be as productive as they choose.

We decide to cut our losses and bounce the server - in the past this has meant a 5 minute outage for everyone but after this all users can get onto the system. This time round however, the system won't remotely reboot. It sits there halfway through the reboot process and becomes completely unresponsive. Luckily we have a colleague down at the remote site with the server and it is forced to shut down and reboot. This does not make it happy! Over the next few hours it has to be repeatedly forced to shutdown until eventually it seems to catch up with itself and goes down gracefully. Luckily it comes back up too!

At this point, we also discovered issues with Terminal Server on the application server - it would not show any TS details and when it did load (which was infrequently) it would not show any license servers at all. We decided to reinstall TS on this server. This went without too much of a hitch but did not fix the Citrix issues.

Citrix was now showing the '*published application* failed to start' error. Little info on the web regarding this, so we ended up doing the following to register all the dll files in the webem folder in the system32 folder:
cmd> CD c:\windows\system32\webem
>for %s IN (*.dll) DO regsvr32 /s %s
Followed by:
>winmgmt /resetrepository

This allowed all the terminal server services to come back online and for us to examine and confirm that this at least was back online.

Finally, we had to make some amendments to the Registry - running regedit we found that the following key was missing some info:
HKLM\Software\Microsoft\WindowsNT\CurrentVersion\WinLogon
the AppSetup key was showing UsrLogon.cmd
but needed to show:
CtxHide.exe UsrLogon.cmd,cmstart.exe

Once these changes had been made, Citrix was back up and running again! Now to do some clean-up routines and find out what exactly went wrong in the first place!

DNS Corrupt

Today I came into work to find that none of our users could access their mailboxes. On inspection, I managed to isolate the problem as a DNS issue. When I looked at the Standard Primary DNS Zone on our Domain Controller I was shown a nice big red cross and "The DNS server encountered a problem while attempting to load the zone. The transfer of zone data from the master server failed."Most forums on the Internet give lots of info on secondary zones and how to reconfigure them to reload from the primary zone - in this case the error was with the Standard Primary Zone and as such, no reloading was possible! For some reason, the zone wasn't set up as an Active Directory Integrated (ADI) zone and as such there was no backup of the zone in AD. As soon as I get a chance, this will be rectified and the zone will become an ADI Zone!

In the meantime - how do we go about solving this problem? I looked in the system32\dns folder and opened the domain.local file to see that it was completely empty - this is where our problem is. If the zone is not ADI, this text file is where all the DNS information is stored. There should also be a Backup folder in here, with a copy of all the files. In my case there was a fully populated domain.local file which I was able to copy back into the system32\dns folder and replace the empty domain.local file. A quick restart of the DNS Server service on the server and Hey Presto! DNS is back up and running!

The Exchange server also required a reboot to bring all the services back up and allow all staff members to access the mailboxes once again.

Moral of the story: Make sure your DNS Zone is ADI and backup the system32\dns folder daily!