I'm copying and pasting parts of it below, simplifying parts, and adding my own hints. This presumes Windows Server 2008. Some Windows 2012 steps are here.
Part 1 - Add IIS if not already installed
- In Server Manager, select Add Roles.
- On the Select Server Roles page, select Web Server (IIS) and select Install.
- Select Next until you get to the Select Role Services page.
- In addition to what is already selected, make sure that ODBC Logging, IIS Metabase Compatibility, and IIS 6 Management Console are selected and then select Next.
- When you’re prompted to install IIS, select Install. You may need to restart the server after the installation is finished.
Part 2 - Install SMTP
- Open Server Manager and select Add Features.
- On the Select Features screen, choose SMTP Server. You may be prompted to install additional components. If that’s the case, select Add Required Features and select Next.
- Select Install. After the installation is finished, you may have to start the SMTP service by using the Services snap-in for the Microsoft Management Console (MMC).
- Office 365 requires TLS encryption and for this server to use TLS, it must have a certificate installed.
- In order to do this the Web Server (IIS) role and IIS Management Console must be installed (needs to be added via Server Manager -> Add Roles).
- To create the self-signed certificate: (Start->Administrative Tools->Internet Information Services (IIS) Manager->Select Host->Server Certificates->Create Self-Signed Certificate)
Part 4 - Configure SMTP server relay
- Start->Administrative Tools->Internet Information Services (IIS) 6.0 Manager.
- Click on the ‘+’ next to your host name.
- Right-click on the [SMTP Virtual Server…] and select Properties. It’s now time to step through each of the tabs to configure the SMTP relay.
- General Tab: The IP address should be set to (All Unassigned)
- Access Tab: Click Authentication… and select the Anonymous access check box.
- Access Tab: Click Connection… Select ‘All Except the list below’ and leave the list below blank. This allows any device inside your firewall to access this relay.
- Access Tab: Click Relay… Select ‘All Except the list below’ and leave the list below blank. This allows any device inside your firewall to access this relay.
- Messages Tab: No changes. The default works well.
- Delivery Tab: Click Outbound Security… Select Basic authentication and enter the username and password that is used to send e-mail to the external server (Office 365 in this case). The user name must be a fully qualified (ex: email@example.com) valid Office 365 user licensed for Exchange. Check TLS encryption.
- Delivery Tab: Click Outbound connections… Set the TCP port to 587.
- Delivery Tab: Click Advanced Delivery and set the Fully-qualified domain name box to the name of the local server that is acting as the relay (ex: myserver1.domain.local). Set the Smart host the full-qualified name of the Office 365 SMTP Server (as of 8/6/14 - this is smtp.office365.com in all cases). Make sure the “Attempt direct…” box is unchecked.
- LDAP Routing and Security Tabs: No changes to these areas.
- Now there has to be a remote domain setup with the Office 365 domain name in it. Click the ‘+’ next to the [SMTP Virtual Server…] item
- Right-click on Domains and select New-Domain which will launch a Wizard.
- Select Remote and Next.
- Enter the name of the Office 365 vanity domain (ex: mycompany.com)
- Now this remote domain will be setup very similarly to the overall SMTP server. Right-click on the new domain name and select Properties.
- Select Forward all mail to smart host and enter the same Office 365 SMTP Server as above (ex. smtp.office365.com)
- Click on Outbound Security and configure the same as above. Select Basic authentication and enter the username and password that is used to send e-mail to the external server (Office 365 in this case). The user name must be a fully qualified (ex: firstname.lastname@example.org) valid Office 365 user licensed for Exchange. Check TLS encryption
- Repeat steps 14 through 19 where step 16 is *.com for the domain
- Repeat steps 14 through 19 where step 16 is *.org for the domain
Part 5 - Configure the on-premise device
- Configure the on-premise device (copier, phone system, etc) with the IP address for the Windows server you have been working with as the SMTP server
- For email address, enter the same address you entered in Part 4 step 9
- Use port 25 and no authentication of any kind and no SSL or any other kind of encryption
- Make sure the firewall on the Windows server allows connections on port 25. A good test is "telnet 10.0.0.18 25" where 10.0.0.18 might be the IP address of the server you're using as the relay
- I've seen instances where the first emails can take up to 90 minutes to relay. I cannot explain this. But it is true.
- As a test, try using Windows Mail or Outlook as a test mechanism. If it succeeds through your test program, it's just a matter of configuring your device (copier, etc) properly
- On the relay server, there can be error messages located here if things aren't coming through after 90 minutes - C:\inetpub\mailroot\Badmail