Logit and Sigmoid Functions and its Calculus

Logit and Sigmoid Functions and its Calculus

Definitions

The sigmoid function is defined as:

sigmoid(x) = 1 ÷ (1 + e^(-x))

The logit function is defined as:  

logit(p) = ln (p ÷ (1 - p))

For logit(p) to have a real number answer, 0 ≤ p < 1

Transform from the Sigmoid Function to the Logit Function

We can easily transform from the sigmoid function to the logit function.  

Let s = sigmoid(x). Then:

s = 1 ÷ (1 + e^(-x))

s * (1 + e^(-x)) = 1

s + s * e^(-x) = 1

s * e^(-x) = 1 - s

e^(-x) = (1 - s) ÷ s

e^x = s ÷ (1 - s)

x = ln(s ÷ (1 - s)) = logit(s)

To transform from the logit function to the sigmoid function, just go backwards.  

Sigmoid Function:  Derivative and Integral

Derivative

s = sigmoid(x)

s = 1 ÷ (1 + e^(-x))

Using the quotient rule of derivatives:

ds/dx = [(1 + e^(-x)) * 0 - 1 * -e^(-x)] ÷ (1 + e^(-x))^2

= -(-e^(-x)) ÷ (1 + e^(-x))^2

= -e^(-x) ÷ (1 + e^(-x))^2

Integral

s = sigmoid(x)

s = 1 ÷ (1 + e^(-x))

Multiply both sides by e^x ÷ e^x:

s * (e^x ÷ e^x) = (e^x ÷ e^x) * (1 ÷ (1 + e^(-x)))

s = e^x ÷ (e^x + 1)

Integral:

∫ e^x ÷ (e^x + 1) dx

Let u = e^x + 1.  Then du = e^x dx 

= ∫  du ÷ (u + 1) 

= ln (u + 1) + C

= ln (e^x + 1) + C

Summary:

d/dx sigmoid(x) = -e^(-x) ÷ (1 + e^(-x))^2

∫ sigmoid(x) dx = ln (e^x + 1) + C

Logit Function:  Derivative and Integral

Derivative

logit(p) = ln (p ÷ (1 - p))

L = ln (p ÷ (1 - p))

Derivative:

dL/dp =  [(1 - p) ÷ p] * d/dp ln (p ÷ (1 - p))

=  [(1 - p) ÷ p] * [(1 - p) * 1 - p * (-1)] ÷ [(1 - p)^2] 

=  [(1 - p) ÷ p] * [1 - p + p] ÷ [(1 - p)^2]

=  [(1 - p) ÷ p] * 1 ÷ (1 - p)^2

= 1 ÷ [p * (1 - p)]

Integral:

∫ ln (p ÷ (1 - p)) dp

By integration by parts:

u = ln (p ÷ (1 - p)) 

du = 1 ÷ [p * (1 - p)] dp

v = dp

v = p

Then:

∫u dv

= p * ln ( p ÷ (1 - p)) - ∫ p ÷ (1 - p) dp

= p * ln ( p ÷ (1 - p)) + ∫ -p ÷ (1 - p) dp

= p * ln ( p ÷ (1 - p)) + ln(1 - p) + C

In Summary:

d/dp logit(p) = 1 ÷ [p * (1 - p)]

∫ logit(p) dp = p * ln ( p ÷ (1 - p)) + ln(1 - p) + C

Eddie

All original content copyright, © 2011-2022.  Edward Shore.   Unauthorized use and/or unauthorized distribution for commercial purposes without express and written permission from the author is strictly prohibited.  This blog entry may be distributed for noncommercial purposes, provided that full credit is given to the author. 

Python - Lambda Week: Derivatives and Newton's Method

Python - Lambda Week: Derivatives and Newton's Method

Welcome to Python Week!  This we we're going to cover calculus and the keyword lambda.

Note:  All Python scripts presented this week were created using a TI-NSpire CX II CAS.   As of June 2022, the lambda keyword is available on all calculators (in the United States) that have Python.   If you are not sure, please check your calculator manual. 

Derivative

The Five Stencil Method is used.  Due to the approximate nature, results are rounded to 5 digits.

f'(x) ≈ (-f(x+2*h) + 8*f(x+h) - 8*f(x-h) + f(x-2*h)) / (12 * h)

h is set to 0.0001 to allow for a wide range of functions and to hopefully prevent float point overflows or underflows.  You can modify h or have the user input a value if you so wish.  

derivlam.py:  Derivative Using the Five Stencil Method

# Math Calculations

#================================

from math import *

#================================

print("The math module is imported.")

f=eval("lambda x:"+input("f(x)? "))

# input x0

x=eval(input("d/dx at x0: "))

h=.0001

# derivative, 5 stencil

d=(-f(x+2*h)+8*f(x+h)-8*f(x-h)+f(x-2*h))/(12*h)

print("round to 5 decimal points")

print("d/dx = "+str(round(d,5)))

Newton's Method

The next script finds the root of f(x) (solve f(x) = 0) with a guess.  

x_n+1 = x_n - f(x_n) / f'(x_n)

The derivative is calculated using the Five Stencil Method.   

I put a limit of 100 iterations because Newton's Method is not always perfect nor this script finds solutions in the complex plane, just the real numbers.  

newtonlam.py

# Math Calculations

#================================

from math import *

#================================

print("The math module is imported.")

print("Solve f(x)=0 to 6 places")

f=eval("lambda x:"+input("f(x)? "))

# input x0

x=eval(input("Guess? "))

h=.0001

w=1

n=1

while fabs(w)>10**(-7):

  d=(-f(x+2*h)+8*f(x+h)-8*f(x-h)+f(x-2*h))/(12*h)

  w=f(x)/d

  x-=w

  n+=1

  if n>100:

    print("iterations exceeded")

    break

if n<101:

  print("x = "+str(round(x,6)))

  print("iterations used: "+str(n))

All original content copyright, © 2011-2022.  Edward Shore.   Unauthorized use and/or unauthorized distribution for commercial purposes without express and written permission from the author is strictly prohibited.  This blog entry may be distributed for noncommercial purposes, provided that full credit is given to the author. 

TI-58/TI-59 Week: Numerical Derivative

TI-58/TI-59 Week:  Numerical Derivative

Introduction

The program estimates the numeric derivative:

d/dx f(x) = (f(x + h) - f(x)) / h

Registers used:

R01:  x

R02:  h

R03:  d/dx

Labels:

[ A ]:  store x

[ B ]:  store h

[ C ]:  calculate derivative

LBL E:  store f(x), assume x is in the display.  You can use registers R00 and R06 and above for registers.  Use Rad  ( [ 2nd ] [ - ] ) if f(x) contains trigonometric functions.  End each function with the steps =, INV SBR (RTN)

Function Listings

(step number, key code, key)

000 76 LBL

001 11 A

002 42 STO

003 01 01

004 92 INV SBR (RTN)

005 76 LBL

006 12 B

007 42 STO

008 02 02

009 92 INV SBR (RTN)

010 76 LBL

011 13 C

012 43 RCL

013 01 01

014 85 +

016 43 RCL

017 02 02

018 71 SBR

019 15 E

020 42 STO

021 03 03

022 43 RCL

023 01 01

024 71 SBR

025 15 E

026 94 +/-

027 85 +

028 43 RCL

029 03 03

030 95 =

031 55 ÷

032 43 RCL

033 02 02

034 95 =

035 42 STO

036 03 03

037 92 INV SBR (RTN)

038 76 LBL

039 15 E

...      .... ....

n-1 95 =

nnn 92 INV SBR (RTN)

Examples

f(x) = sin x;    Rad, sin

x = 0.5, h = 0.1, [ C ] returns 0.8521693479

x = 0.5, h = 1E-8, [ C ] returns 0.87755

f(x) = (1 + cos x)^1.5;   Rad, cos, +, 1, =, y^x, 1.5

x = 2, h = 0.01, [ C ] returns -1.035743295

x = 2, h = 1E-8, [ C ] returns -1.04206

f(x) = x * e^(x);   STO, 00, e^x * RCL, 00

x = 3, h = 0.01, [ C ] returns 80.84630054

x = 3, h = 1E-5, [ C ] returns 80.342648

Eddie

All original content copyright, © 2011-2022.  Edward Shore.   Unauthorized use and/or unauthorized distribution for commercial purposes without express and written permission from the author is strictly prohibited.  This blog entry may be distributed for noncommercial purposes, provided that full credit is given to the author.